{"active_field":{"id":"otb-20261110T173446Z","engine_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","started_at":"2026-11-10T17:34:46Z","gateways":["agentgateway","aisix","apisix","bifrost","busbar","gomodel","helicone","higress","kong","litellm-python","litellm-rust","one-api","plano","portkey","tensorzero"]},"schema_version":"1.0","collection_kind":"independent_gateway_runs","measurements":[{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:openai-responses:openai-responses","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":210,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":247,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":277,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":195.8575912414804,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5105.750528541226,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9823738537057497,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":120751,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":20122,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.4609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":53.240234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":50.75390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":54.40625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":54.2421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.763626123026112,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.031671412126784065,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":19964,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":600,"first_disqualified_conc":16,"lower_bound":false,"samples":119803},{"bound_ms":5,"rps":{"value":20925,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":4840,"first_disqualified_conc":128,"lower_bound":false,"samples":125554},{"bound_ms":10,"rps":{"value":21051,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":9950,"first_disqualified_conc":256,"lower_bound":false,"samples":126327},{"bound_ms":50,"rps":{"value":21224,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":23552,"first_disqualified_conc":1024,"lower_bound":false,"samples":127359},{"bound_ms":100,"rps":{"value":21224,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":23552,"first_disqualified_conc":2048,"lower_bound":false,"samples":127359},{"bound_ms":null,"rps":{"value":21224,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":23552,"first_disqualified_conc":null,"lower_bound":false,"samples":127359}],"sample_counts":{"gateway_c1_samples":25353,"direct_c1_samples":241872},"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:cohere:openai-responses","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:openai:openai-responses","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:anthropic:openai-responses","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":227,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":267,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":299,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":32,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":207.76145582109186,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":4813.212325875897,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9840677551204744,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":114025,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":19002,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.33984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":54.75,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":51.7734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":55.41796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":55.16015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":252,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.175601960927026,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.04876842506033785,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":18964,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":623,"first_disqualified_conc":16,"lower_bound":false,"samples":113798},{"bound_ms":5,"rps":{"value":19692,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":2538,"first_disqualified_conc":64,"lower_bound":false,"samples":118168},{"bound_ms":10,"rps":{"value":19772,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":5124,"first_disqualified_conc":128,"lower_bound":false,"samples":118635},{"bound_ms":50,"rps":{"value":20024,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24981,"first_disqualified_conc":1024,"lower_bound":false,"samples":120161},{"bound_ms":100,"rps":{"value":20024,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24981,"first_disqualified_conc":2048,"lower_bound":false,"samples":120161},{"bound_ms":null,"rps":{"value":20024,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24981,"first_disqualified_conc":null,"lower_bound":false,"samples":120161}],"sample_counts":{"gateway_c1_samples":23610,"direct_c1_samples":229535},"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:bedrock:openai-responses","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:gemini:openai-responses","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:anthropic:bedrock","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":218,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":256,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":287,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":31,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":200.20825044232055,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":4994.799154334039,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9823076767856654,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":118127,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":19685,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.07421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":54.66015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":51.2421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":55.44921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":55.2578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still falling - releasing memory, not leaking it when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-1.302289476054103,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.04872452430819797,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":19560,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":617,"first_disqualified_conc":16,"lower_bound":false,"samples":117376},{"bound_ms":5,"rps":{"value":20404,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":4993,"first_disqualified_conc":128,"lower_bound":false,"samples":122433},{"bound_ms":10,"rps":{"value":20404,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":4993,"first_disqualified_conc":128,"lower_bound":false,"samples":122433},{"bound_ms":50,"rps":{"value":20660,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24142,"first_disqualified_conc":1024,"lower_bound":false,"samples":123971},{"bound_ms":100,"rps":{"value":20660,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24142,"first_disqualified_conc":2048,"lower_bound":false,"samples":123971},{"bound_ms":null,"rps":{"value":20660,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24142,"first_disqualified_conc":null,"lower_bound":false,"samples":123971}],"sample_counts":{"gateway_c1_samples":24390,"direct_c1_samples":237655},"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:gemini:bedrock","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:cohere:bedrock","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:openai-responses:bedrock","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":231,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":270,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":31,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":208.7132725430598,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":4791.262135922329,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9840643588611752,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":113505,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":18915,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.09765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":54.751953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":51.9921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":55.734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":55.29296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.02300161895283871,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.014420287700335939,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":18692,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":633,"first_disqualified_conc":16,"lower_bound":false,"samples":112168},{"bound_ms":5,"rps":{"value":19412,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":2579,"first_disqualified_conc":64,"lower_bound":false,"samples":116482},{"bound_ms":10,"rps":{"value":19442,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":5234,"first_disqualified_conc":128,"lower_bound":false,"samples":116658},{"bound_ms":50,"rps":{"value":19736,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":25370,"first_disqualified_conc":1024,"lower_bound":false,"samples":118430},{"bound_ms":100,"rps":{"value":19736,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":25370,"first_disqualified_conc":2048,"lower_bound":false,"samples":118430},{"bound_ms":null,"rps":{"value":19736,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":25370,"first_disqualified_conc":null,"lower_bound":false,"samples":118430}],"sample_counts":{"gateway_c1_samples":23233,"direct_c1_samples":241925},"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:bedrock:bedrock","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:openai:bedrock","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":219,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":256,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":286,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":205.61558824805797,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":4863.4444913465595,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9840817944727618,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":115215,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":19200,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.1953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":53.9375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":50.9140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":54.71875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":54.3125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still falling - releasing memory, not leaking it when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.6369156722067312,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.0445382197398453,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":19159,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":628,"first_disqualified_conc":16,"lower_bound":false,"samples":114971},{"bound_ms":5,"rps":{"value":19912,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":2516,"first_disqualified_conc":64,"lower_bound":false,"samples":119487},{"bound_ms":10,"rps":{"value":20007,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":5099,"first_disqualified_conc":128,"lower_bound":false,"samples":120063},{"bound_ms":50,"rps":{"value":20265,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24671,"first_disqualified_conc":1024,"lower_bound":false,"samples":121609},{"bound_ms":100,"rps":{"value":20265,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24671,"first_disqualified_conc":2048,"lower_bound":false,"samples":121609},{"bound_ms":null,"rps":{"value":20265,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24671,"first_disqualified_conc":null,"lower_bound":false,"samples":121609}],"sample_counts":{"gateway_c1_samples":24338,"direct_c1_samples":241582},"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:anthropic:cohere","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:openai-responses:cohere","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:gemini:cohere","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:cohere:cohere","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:openai:cohere","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:bedrock:cohere","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:openai-responses:openai","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":229,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":267,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":298,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":31,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":207.54121914039914,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":4818.319966230477,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9840504611107183,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":114146,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":19022,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.03515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":53.9375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":51.01171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":54.8046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":54.5703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":1.1853562210621402,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.045130039098198085,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":19011,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":624,"first_disqualified_conc":16,"lower_bound":false,"samples":114080},{"bound_ms":5,"rps":{"value":19731,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":2536,"first_disqualified_conc":64,"lower_bound":false,"samples":118399},{"bound_ms":10,"rps":{"value":19868,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":5119,"first_disqualified_conc":128,"lower_bound":false,"samples":119220},{"bound_ms":50,"rps":{"value":20063,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24817,"first_disqualified_conc":1024,"lower_bound":false,"samples":120396},{"bound_ms":100,"rps":{"value":20063,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24817,"first_disqualified_conc":2048,"lower_bound":false,"samples":120396},{"bound_ms":null,"rps":{"value":20063,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24817,"first_disqualified_conc":null,"lower_bound":false,"samples":120396}],"sample_counts":{"gateway_c1_samples":23446,"direct_c1_samples":240137},"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:cohere:openai","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:openai:openai","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":216,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":251,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":282,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":31,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":188.57584939720198,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5302.905982905983,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9718871854587116,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":124088,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":20679,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":327,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":360,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21155us) came in under the mock's (21182us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":32614.06218906021,"certified":true,"suppressed":false,"headroom":0.3135201901108999,"rig_ceiling":104025.39682539682,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":2048,"certified":true,"suppressed":false,"headroom":0.3135201901108999,"rig_ceiling":104025.39682539682,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.00390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":53.4921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":49.99609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":54.5234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":54.2265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still falling - releasing memory, not leaking it when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-2.0749110660555545,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.03612607648971414,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":21156,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":588,"first_disqualified_conc":16,"lower_bound":false,"samples":126952},{"bound_ms":5,"rps":{"value":22239,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":4565,"first_disqualified_conc":128,"lower_bound":false,"samples":133447},{"bound_ms":10,"rps":{"value":22380,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":9410,"first_disqualified_conc":256,"lower_bound":false,"samples":134304},{"bound_ms":50,"rps":{"value":22545,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":22674,"first_disqualified_conc":1024,"lower_bound":false,"samples":135283},{"bound_ms":100,"rps":{"value":22545,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":22674,"first_disqualified_conc":2048,"lower_bound":false,"samples":135283},{"bound_ms":null,"rps":{"value":22545,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":22674,"first_disqualified_conc":null,"lower_bound":false,"samples":135283}],"sample_counts":{"gateway_c1_samples":24618,"direct_c1_samples":233879,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":4096,"http_status_counts":{"200":4096},"stream_errors_no_response":0},{"conc":3072,"http_status_counts":{"200":3072},"stream_errors_no_response":0},{"conc":2560,"http_status_counts":{"200":2560},"stream_errors_no_response":0},{"conc":2304,"http_status_counts":{"200":2304},"stream_errors_no_response":0},{"conc":2432,"http_status_counts":{"200":2432},"stream_errors_no_response":0},{"conc":2496,"http_status_counts":{"200":2496},"stream_errors_no_response":0},{"conc":2528,"http_status_counts":{"200":2528},"stream_errors_no_response":0},{"conc":2544,"http_status_counts":{"200":2544},"stream_errors_no_response":0},{"conc":2552,"http_status_counts":{"200":2552},"stream_errors_no_response":0},{"conc":2556,"http_status_counts":{"200":2556},"stream_errors_no_response":0},{"conc":2558,"http_status_counts":{"200":2558},"stream_errors_no_response":0},{"conc":2557,"http_status_counts":{"200":2557},"stream_errors_no_response":0},{"conc":2556,"http_status_counts":{"200":2556},"stream_errors_no_response":0},{"conc":2556,"http_status_counts":{"200":2556},"stream_errors_no_response":0},{"conc":2302,"http_status_counts":{"200":2302},"stream_errors_no_response":0},{"conc":2175,"http_status_counts":{"200":2175},"stream_errors_no_response":0},{"conc":2111,"http_status_counts":{"200":2111},"stream_errors_no_response":0},{"conc":2079,"http_status_counts":{"200":2079},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:anthropic:openai","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":212,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":247,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":279,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":32,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":196.34930127951347,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5092.9643929643935,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9681526005189147,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":118717,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":19784,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":341,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":395,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":3,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":17,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"the bisection proved c=4128, but it did not hold on re-measurement and the stepped-down rung at c=4098 failed the stream gate on its first window","state":"not_measured","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"the bisection proved c=4128, but it did not hold on re-measurement and the stepped-down rung at c=4098 failed the stream gate on its first window","state":"not_measured","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":33.6953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":53.57421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":49.90234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":54.2109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":54.0546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":156,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.10023099694236136,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.04929642550447049,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":19933,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":601,"first_disqualified_conc":16,"lower_bound":false,"samples":119616},{"bound_ms":5,"rps":{"value":20841,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":4866,"first_disqualified_conc":128,"lower_bound":false,"samples":125053},{"bound_ms":10,"rps":{"value":20997,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":9889,"first_disqualified_conc":256,"lower_bound":false,"samples":126001},{"bound_ms":50,"rps":{"value":21100,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":23269,"first_disqualified_conc":1024,"lower_bound":false,"samples":126618},{"bound_ms":100,"rps":{"value":21100,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":23269,"first_disqualified_conc":2048,"lower_bound":false,"samples":126618},{"bound_ms":null,"rps":{"value":21100,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":23269,"first_disqualified_conc":null,"lower_bound":false,"samples":126618}],"sample_counts":{"gateway_c1_samples":25009,"direct_c1_samples":233282,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":4096,"http_status_counts":{"200":4096},"stream_errors_no_response":0},{"conc":8192,"http_status_counts":{"200":8192},"stream_errors_no_response":0},{"conc":6144,"http_status_counts":{"200":6144},"stream_errors_no_response":0},{"conc":5120,"http_status_counts":{"200":5120},"stream_errors_no_response":0},{"conc":4608,"http_status_counts":{"200":4608},"stream_errors_no_response":0},{"conc":4352,"http_status_counts":{"200":4352},"stream_errors_no_response":0},{"conc":4224,"http_status_counts":{"200":4224},"stream_errors_no_response":0},{"conc":4160,"http_status_counts":{"200":4160},"stream_errors_no_response":0},{"conc":4128,"http_status_counts":{"200":4128},"stream_errors_no_response":0},{"conc":4144,"http_status_counts":{"200":4144},"stream_errors_no_response":0},{"conc":4136,"http_status_counts":{"200":3945},"stream_errors_no_response":191},{"conc":4132,"http_status_counts":{"200":3959},"stream_errors_no_response":173},{"conc":4130,"http_status_counts":{"200":4130},"stream_errors_no_response":0},{"conc":4129,"http_status_counts":{"200":4129},"stream_errors_no_response":0},{"conc":4128,"http_status_counts":{"200":3234},"stream_errors_no_response":894},{"conc":4128,"http_status_counts":{"200":4057},"stream_errors_no_response":71},{"conc":4112,"http_status_counts":{"200":3242},"stream_errors_no_response":870},{"conc":4104,"http_status_counts":{"200":2957},"stream_errors_no_response":1147},{"conc":4100,"http_status_counts":{"200":3860},"stream_errors_no_response":240},{"conc":4098,"http_status_counts":{"200":2704},"stream_errors_no_response":1394},{"conc":4096,"http_status_counts":{"200":3404},"stream_errors_no_response":692},{"conc":4096,"http_status_counts":{"200":3208},"stream_errors_no_response":888},{"conc":4096,"http_status_counts":{"200":3250},"stream_errors_no_response":846}],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:bedrock:openai","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:gemini:openai","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:anthropic:anthropic","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":206,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":240,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":271,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":31,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":191.31926970324474,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5226.865028029322,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.963164850868556,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":121211,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":20199,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":323,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":362,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":3,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21129us) came in under the mock's (21145us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":29596.665864916016,"certified":true,"suppressed":false,"headroom":0.14197961969189427,"rig_ceiling":208457.14285714287,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":4104,"certified":true,"suppressed":false,"headroom":0.14197961969189427,"rig_ceiling":208457.14285714287,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.15625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":53.16796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":50.18359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":54.41015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":54.24609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.23727509558911336,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.048736586694404796,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":20502,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":614,"first_disqualified_conc":16,"lower_bound":false,"samples":123033},{"bound_ms":5,"rps":{"value":21665,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":4706,"first_disqualified_conc":128,"lower_bound":false,"samples":130000},{"bound_ms":10,"rps":{"value":21795,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":9616,"first_disqualified_conc":256,"lower_bound":false,"samples":130775},{"bound_ms":50,"rps":{"value":21982,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":22671,"first_disqualified_conc":1024,"lower_bound":false,"samples":131913},{"bound_ms":100,"rps":{"value":21982,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":22671,"first_disqualified_conc":2048,"lower_bound":false,"samples":131913},{"bound_ms":null,"rps":{"value":21982,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":22671,"first_disqualified_conc":null,"lower_bound":false,"samples":131913}],"sample_counts":{"gateway_c1_samples":25747,"direct_c1_samples":233859,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":4096,"http_status_counts":{"200":4096},"stream_errors_no_response":0},{"conc":8192,"http_status_counts":{"200":8192},"stream_errors_no_response":0},{"conc":6144,"http_status_counts":{"200":6144},"stream_errors_no_response":0},{"conc":5120,"http_status_counts":{"200":5120},"stream_errors_no_response":0},{"conc":4608,"http_status_counts":{"200":4608},"stream_errors_no_response":0},{"conc":4352,"http_status_counts":{"200":4352},"stream_errors_no_response":0},{"conc":4224,"http_status_counts":{"200":4224},"stream_errors_no_response":0},{"conc":4160,"http_status_counts":{"200":4160},"stream_errors_no_response":0},{"conc":4128,"http_status_counts":{"200":4128},"stream_errors_no_response":0},{"conc":4112,"http_status_counts":{"200":4112},"stream_errors_no_response":0},{"conc":4104,"http_status_counts":{"200":4104},"stream_errors_no_response":0},{"conc":4108,"http_status_counts":{"200":4108},"stream_errors_no_response":0},{"conc":4106,"http_status_counts":{"200":4106},"stream_errors_no_response":0},{"conc":4105,"http_status_counts":{"200":4105},"stream_errors_no_response":0},{"conc":4104,"http_status_counts":{"200":4104},"stream_errors_no_response":0},{"conc":4104,"http_status_counts":{"200":3923},"stream_errors_no_response":181}],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:bedrock:anthropic","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:openai-responses:anthropic","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:openai:anthropic","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":228,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":268,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":299,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":31,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":206.99208794721747,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":4831.102531102531,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9682216923673016,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":112613,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":18767,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":405,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":446,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":6,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":22,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":18079.394815032054,"certified":true,"suppressed":false,"headroom":0.13909264768305726,"rig_ceiling":129980.95238095238,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":2559,"certified":true,"suppressed":false,"headroom":0.13909264768305726,"rig_ceiling":129980.95238095238,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.1953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":53.81640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":50.52734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":54.7109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":54.3515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.7642133230320394,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.04117718673808115,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":19127,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":620,"first_disqualified_conc":16,"lower_bound":false,"samples":114783},{"bound_ms":5,"rps":{"value":19852,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":2527,"first_disqualified_conc":64,"lower_bound":false,"samples":119122},{"bound_ms":10,"rps":{"value":19973,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":5072,"first_disqualified_conc":128,"lower_bound":false,"samples":119864},{"bound_ms":50,"rps":{"value":20228,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24787,"first_disqualified_conc":1024,"lower_bound":false,"samples":121382},{"bound_ms":100,"rps":{"value":20228,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24787,"first_disqualified_conc":2048,"lower_bound":false,"samples":121382},{"bound_ms":null,"rps":{"value":20228,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24787,"first_disqualified_conc":null,"lower_bound":false,"samples":121382}],"sample_counts":{"gateway_c1_samples":23411,"direct_c1_samples":237919,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":4096,"http_status_counts":{"200":4096},"stream_errors_no_response":0},{"conc":3072,"http_status_counts":{"200":3072},"stream_errors_no_response":0},{"conc":2560,"http_status_counts":{"200":2560},"stream_errors_no_response":0},{"conc":2304,"http_status_counts":{"200":2304},"stream_errors_no_response":0},{"conc":2432,"http_status_counts":{"200":2432},"stream_errors_no_response":0},{"conc":2496,"http_status_counts":{"200":2496},"stream_errors_no_response":0},{"conc":2528,"http_status_counts":{"200":2528},"stream_errors_no_response":0},{"conc":2544,"http_status_counts":{"200":2544},"stream_errors_no_response":0},{"conc":2552,"http_status_counts":{"200":2552},"stream_errors_no_response":0},{"conc":2556,"http_status_counts":{"200":2556},"stream_errors_no_response":0},{"conc":2558,"http_status_counts":{"200":2558},"stream_errors_no_response":0},{"conc":2559,"http_status_counts":{"200":2559},"stream_errors_no_response":0},{"conc":2559,"http_status_counts":{"200":2559},"stream_errors_no_response":0},{"conc":2559,"http_status_counts":{"200":2559},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:cohere:anthropic","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:gemini:anthropic","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:cohere:gemini","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:gemini:gemini","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":211,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":248,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":278,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":196.89006971107798,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5078.976311336717,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9820473628139751,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":120067,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":20008,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":33.95703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":54.23828125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":50.96484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":54.76953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":54.5234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.1775921183649501,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.041368822953469664,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":19915,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":613,"first_disqualified_conc":16,"lower_bound":false,"samples":119508},{"bound_ms":5,"rps":{"value":20912,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":4839,"first_disqualified_conc":128,"lower_bound":false,"samples":125481},{"bound_ms":10,"rps":{"value":20976,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":9978,"first_disqualified_conc":256,"lower_bound":false,"samples":125877},{"bound_ms":50,"rps":{"value":21168,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":23722,"first_disqualified_conc":1024,"lower_bound":false,"samples":127023},{"bound_ms":100,"rps":{"value":21168,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":23722,"first_disqualified_conc":2048,"lower_bound":false,"samples":127023},{"bound_ms":null,"rps":{"value":21168,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":23722,"first_disqualified_conc":null,"lower_bound":false,"samples":127023}],"sample_counts":{"gateway_c1_samples":25203,"direct_c1_samples":241373},"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:openai-responses:gemini","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:openai:gemini","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":217,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":253,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":283,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":202.1159412864718,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":4947.65525982256,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9832975723418331,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":117111,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":19516,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.05859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":53.58984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":50.46875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":54.25,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":53.984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.36093857733091694,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.0024901789418739434,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":19532,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":613,"first_disqualified_conc":16,"lower_bound":false,"samples":117207},{"bound_ms":5,"rps":{"value":20455,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":4966,"first_disqualified_conc":128,"lower_bound":false,"samples":122740},{"bound_ms":10,"rps":{"value":20455,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":4966,"first_disqualified_conc":128,"lower_bound":false,"samples":122740},{"bound_ms":50,"rps":{"value":20723,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24131,"first_disqualified_conc":1024,"lower_bound":false,"samples":124350},{"bound_ms":100,"rps":{"value":20723,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24131,"first_disqualified_conc":2048,"lower_bound":false,"samples":124350},{"bound_ms":null,"rps":{"value":20723,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":24131,"first_disqualified_conc":null,"lower_bound":false,"samples":124350}],"sample_counts":{"gateway_c1_samples":24639,"direct_c1_samples":244550},"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:bedrock:gemini","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c:anthropic:gemini","gateway_id":"agentgateway","gateway_build":"ghcr.io/agentgateway/agentgateway:v1.7.0@sha256:4562330cd6e580bfac2de5b8905fee608dcf3fff6f1f4900744d27bdddb3b41b","run_id":"run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","measured_at":"2026-11-10T17:39:19Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","detail":"Pinned v1.7.0 image and source 26f5aebe99a36a037fe54f8080a608aca5547e15: CHAT_TRANSLATIONS in crates/agentgateway/src/llm/mod.rs defines supported conversions. Separate custom providers constrain Chat versus Responses wire. Gemini native and Chat-to-Gemini are supported; Messages/Responses use Google OpenAI compatibility instead of native Gemini and are not counted as Gemini wire. Cohere preset similarly uses OpenAI compatibility, not Cohere native chat. No native Cohere or Bedrock ingress conversion is registered. Local recorder and response checks establish 12 pairs. Responses mock token-details correction is required for Messages-to-Responses. Evidence: results/diagnostics/refresh-configuration/agentgateway-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-0.8586367841661555,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":95373,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/agentgateway","raw_result_url":"https://onthebench.ai/api/v1/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-agentgateway-fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"fa351efdcee949e977dd1d4331440512b3e913a74b878f4bd1b19a7c7ca2900c"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:openai:anthropic","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":162,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":190,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":218,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":158.5811513058638,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":6305.919661733615,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7859282595133944,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":119308,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":19882,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":16,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.000008381667616588997,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":294,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":500,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":9,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":19,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":34767.76115030459,"certified":true,"suppressed":false,"headroom":0.8832132872859635,"rig_ceiling":39365.079365079364,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":775,"certified":true,"suppressed":false,"headroom":0.8832132872859635,"rig_ceiling":39365.079365079364,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":67.6328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":54.01953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":69.20703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":68.87890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.10804699761897836,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.6871345013915122,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":26114,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":943,"first_disqualified_conc":32,"lower_bound":false,"samples":156705},{"bound_ms":5,"rps":{"value":28484,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":3634,"first_disqualified_conc":128,"lower_bound":false,"samples":170915},{"bound_ms":10,"rps":{"value":29206,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":6753,"first_disqualified_conc":256,"lower_bound":false,"samples":175266},{"bound_ms":50,"rps":{"value":29206,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":6753,"first_disqualified_conc":1024,"lower_bound":false,"samples":175266},{"bound_ms":100,"rps":{"value":29206,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":6753,"first_disqualified_conc":2048,"lower_bound":false,"samples":175266},{"bound_ms":null,"rps":{"value":29206,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":6753,"first_disqualified_conc":null,"lower_bound":false,"samples":175266}],"sample_counts":{"gateway_c1_samples":32039,"direct_c1_samples":261797,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":768,"http_status_counts":{"200":768},"stream_errors_no_response":0},{"conc":896,"http_status_counts":{"200":896},"stream_errors_no_response":0},{"conc":832,"http_status_counts":{"200":832},"stream_errors_no_response":0},{"conc":800,"http_status_counts":{"200":800},"stream_errors_no_response":0},{"conc":784,"http_status_counts":{"200":784},"stream_errors_no_response":0},{"conc":776,"http_status_counts":{"200":776},"stream_errors_no_response":0},{"conc":772,"http_status_counts":{"200":772},"stream_errors_no_response":0},{"conc":774,"http_status_counts":{"200":774},"stream_errors_no_response":0},{"conc":775,"http_status_counts":{"200":775},"stream_errors_no_response":0},{"conc":775,"http_status_counts":{"200":775},"stream_errors_no_response":0},{"conc":775,"http_status_counts":{"200":775},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:bedrock:anthropic","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:gemini:anthropic","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:anthropic:anthropic","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":150,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":175,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":204,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":29,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":150.0564829923434,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":6664.157256378085,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9930564120096029,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":159340,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":26553,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":16,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":271,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":444,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":3,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21138us) came in under the mock's (21217us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":32295.15318055963,"certified":true,"suppressed":false,"headroom":0.9391592736222566,"rig_ceiling":34387.30158730159,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":677,"certified":true,"suppressed":false,"headroom":0.9391592736222566,"rig_ceiling":34387.30158730159,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.68359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":68.68359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":52.7578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":70.72265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":70.5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.3248954130092891,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.6867965671233695,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":28154,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":957,"first_disqualified_conc":32,"lower_bound":false,"samples":168946},{"bound_ms":5,"rps":{"value":30535,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":3505,"first_disqualified_conc":128,"lower_bound":false,"samples":183220},{"bound_ms":10,"rps":{"value":31272,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7495,"first_disqualified_conc":256,"lower_bound":false,"samples":187661},{"bound_ms":50,"rps":{"value":31272,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7495,"first_disqualified_conc":1024,"lower_bound":false,"samples":187661},{"bound_ms":100,"rps":{"value":31272,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7495,"first_disqualified_conc":2048,"lower_bound":false,"samples":187661},{"bound_ms":null,"rps":{"value":31272,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7495,"first_disqualified_conc":null,"lower_bound":false,"samples":187661}],"sample_counts":{"gateway_c1_samples":34173,"direct_c1_samples":257073,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":768,"http_status_counts":{"200":768},"stream_errors_no_response":0},{"conc":896,"http_status_counts":{"200":896},"stream_errors_no_response":0},{"conc":832,"http_status_counts":{"200":832},"stream_errors_no_response":0},{"conc":864,"http_status_counts":{"200":864},"stream_errors_no_response":0},{"conc":848,"http_status_counts":{"200":848},"stream_errors_no_response":0},{"conc":840,"http_status_counts":{"200":840},"stream_errors_no_response":0},{"conc":844,"http_status_counts":{"200":844},"stream_errors_no_response":0},{"conc":842,"http_status_counts":{"200":842},"stream_errors_no_response":0},{"conc":843,"http_status_counts":{"200":843},"stream_errors_no_response":0},{"conc":843,"http_status_counts":{"200":843},"stream_errors_no_response":0},{"conc":843,"http_status_counts":{"200":843},"stream_errors_no_response":0},{"conc":677,"http_status_counts":{"200":677},"stream_errors_no_response":0},{"conc":677,"http_status_counts":{"200":677},"stream_errors_no_response":0},{"conc":677,"http_status_counts":{"200":677},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:openai-responses:anthropic","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":182,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":215,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":244,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":29,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":181.1190625143049,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5521.2299915754,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9860605945290514,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":131074,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":21842,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":16,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.4609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":65.390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":52.29296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":67.2421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":67.08203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.11263145814918507,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.8321574586867803,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":22218,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":539,"first_disqualified_conc":16,"lower_bound":false,"samples":133328},{"bound_ms":5,"rps":{"value":25257,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":4509,"first_disqualified_conc":128,"lower_bound":false,"samples":151548},{"bound_ms":10,"rps":{"value":25869,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":9032,"first_disqualified_conc":256,"lower_bound":false,"samples":155239},{"bound_ms":50,"rps":{"value":25869,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":9032,"first_disqualified_conc":1024,"lower_bound":false,"samples":155239},{"bound_ms":100,"rps":{"value":25869,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":9032,"first_disqualified_conc":2048,"lower_bound":false,"samples":155239},{"bound_ms":null,"rps":{"value":25869,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":9032,"first_disqualified_conc":null,"lower_bound":false,"samples":155239}],"sample_counts":{"gateway_c1_samples":28838,"direct_c1_samples":259566},"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:cohere:anthropic","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:cohere:openai","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:bedrock:openai","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:anthropic:openai","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":152,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":178,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":207,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":29,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":152.09601844195544,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":6574.794069192751,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7563901876709777,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":119727,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":19952,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":16,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":275,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":461,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21066us) came in under the mock's (21074us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21123us) came in under the mock's (21244us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":31294.445560658252,"certified":true,"suppressed":false,"headroom":0.9363364695675674,"rig_ceiling":33422.22222222222,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":658,"certified":true,"suppressed":false,"headroom":0.9363364695675674,"rig_ceiling":33422.22222222222,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.21875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":67.296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":51.80859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":68.921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":68.55859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.5069564534534243,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.9203045680697568,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":27958,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":967,"first_disqualified_conc":32,"lower_bound":false,"samples":167769},{"bound_ms":5,"rps":{"value":30196,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":3454,"first_disqualified_conc":128,"lower_bound":false,"samples":181184},{"bound_ms":10,"rps":{"value":31041,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7584,"first_disqualified_conc":256,"lower_bound":false,"samples":186278},{"bound_ms":50,"rps":{"value":31041,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7584,"first_disqualified_conc":1024,"lower_bound":false,"samples":186278},{"bound_ms":100,"rps":{"value":31041,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7584,"first_disqualified_conc":2048,"lower_bound":false,"samples":186278},{"bound_ms":null,"rps":{"value":31041,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7584,"first_disqualified_conc":null,"lower_bound":false,"samples":186278}],"sample_counts":{"gateway_c1_samples":34038,"direct_c1_samples":258158,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":768,"http_status_counts":{"200":768},"stream_errors_no_response":0},{"conc":896,"http_status_counts":{"200":896},"stream_errors_no_response":0},{"conc":832,"http_status_counts":{"200":832},"stream_errors_no_response":0},{"conc":800,"http_status_counts":{"200":800},"stream_errors_no_response":0},{"conc":816,"http_status_counts":{"200":816},"stream_errors_no_response":0},{"conc":808,"http_status_counts":{"200":808},"stream_errors_no_response":0},{"conc":804,"http_status_counts":{"200":804},"stream_errors_no_response":0},{"conc":806,"http_status_counts":{"200":806},"stream_errors_no_response":0},{"conc":805,"http_status_counts":{"200":805},"stream_errors_no_response":0},{"conc":804,"http_status_counts":{"200":804},"stream_errors_no_response":0},{"conc":804,"http_status_counts":{"200":804},"stream_errors_no_response":0},{"conc":658,"http_status_counts":{"200":658},"stream_errors_no_response":0},{"conc":658,"http_status_counts":{"200":658},"stream_errors_no_response":0},{"conc":658,"http_status_counts":{"200":658},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:gemini:openai","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:openai-responses:openai","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":169,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":199,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":228,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":29,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":162.11809131479575,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":6168.343038644785,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7846584893414216,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":116520,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":19417,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":16,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.46875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":66.85546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":52.69921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":68.234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":68.01171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":1.2399358321787064,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.8168488447601818,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":25449,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":979,"first_disqualified_conc":32,"lower_bound":false,"samples":152712},{"bound_ms":5,"rps":{"value":27472,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":4926,"first_disqualified_conc":128,"lower_bound":false,"samples":164835},{"bound_ms":10,"rps":{"value":28458,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":8386,"first_disqualified_conc":256,"lower_bound":false,"samples":170775},{"bound_ms":50,"rps":{"value":28458,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":8386,"first_disqualified_conc":1024,"lower_bound":false,"samples":170775},{"bound_ms":100,"rps":{"value":28458,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":8386,"first_disqualified_conc":2048,"lower_bound":false,"samples":170775},{"bound_ms":null,"rps":{"value":28458,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":8386,"first_disqualified_conc":null,"lower_bound":false,"samples":170775}],"sample_counts":{"gateway_c1_samples":30866,"direct_c1_samples":260253},"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:openai:openai","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":146,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":174,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":202,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":152.64101169979259,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":6551.319261213721,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9445269209542334,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":148977,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":24826,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":16,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":277,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":366,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":2,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21132us) came in under the mock's (21145us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":30259.205544499786,"certified":true,"suppressed":false,"headroom":0.9236094715617666,"rig_ceiling":32761.904761904763,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":645,"certified":true,"suppressed":false,"headroom":0.9236094715617666,"rig_ceiling":32761.904761904763,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.2890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":68.62890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":52.296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":70.42578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":70.28515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.37583951307297037,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.8534572049737088,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":28888,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":962,"first_disqualified_conc":32,"lower_bound":false,"samples":173346},{"bound_ms":5,"rps":{"value":31374,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":4081,"first_disqualified_conc":128,"lower_bound":false,"samples":188254},{"bound_ms":10,"rps":{"value":32092,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":6596,"first_disqualified_conc":256,"lower_bound":false,"samples":192583},{"bound_ms":50,"rps":{"value":32092,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":6596,"first_disqualified_conc":1024,"lower_bound":false,"samples":192583},{"bound_ms":100,"rps":{"value":32092,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":6596,"first_disqualified_conc":2048,"lower_bound":false,"samples":192583},{"bound_ms":null,"rps":{"value":32092,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":6596,"first_disqualified_conc":null,"lower_bound":false,"samples":192583}],"sample_counts":{"gateway_c1_samples":35048,"direct_c1_samples":262627,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":768,"http_status_counts":{"200":768},"stream_errors_no_response":0},{"conc":896,"http_status_counts":{"200":896},"stream_errors_no_response":0},{"conc":832,"http_status_counts":{"200":832},"stream_errors_no_response":0},{"conc":800,"http_status_counts":{"200":800},"stream_errors_no_response":0},{"conc":784,"http_status_counts":{"200":784},"stream_errors_no_response":0},{"conc":776,"http_status_counts":{"200":776},"stream_errors_no_response":0},{"conc":780,"http_status_counts":{"200":780},"stream_errors_no_response":0},{"conc":778,"http_status_counts":{"200":778},"stream_errors_no_response":0},{"conc":779,"http_status_counts":{"200":779},"stream_errors_no_response":0},{"conc":779,"http_status_counts":{"200":779},"stream_errors_no_response":0},{"conc":779,"http_status_counts":{"200":779},"stream_errors_no_response":0},{"conc":645,"http_status_counts":{"200":645},"stream_errors_no_response":0},{"conc":645,"http_status_counts":{"200":645},"stream_errors_no_response":0},{"conc":645,"http_status_counts":{"200":645},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:openai-responses:cohere","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:openai:cohere","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:bedrock:cohere","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:gemini:cohere","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:cohere:cohere","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:anthropic:cohere","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:anthropic:gemini","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":162,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":187,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":216,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":29,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":163.12134905764356,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":6130.405405405406,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9835725913841037,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":145168,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":24191,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":16,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.71875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":67.51953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":52.1015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":69.29296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":69.109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.6152929252623993,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.7567316178676103,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":26504,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":923,"first_disqualified_conc":32,"lower_bound":false,"samples":159044},{"bound_ms":5,"rps":{"value":28918,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":4780,"first_disqualified_conc":128,"lower_bound":false,"samples":173514},{"bound_ms":10,"rps":{"value":29709,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":8085,"first_disqualified_conc":256,"lower_bound":false,"samples":178283},{"bound_ms":50,"rps":{"value":29709,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":8085,"first_disqualified_conc":1024,"lower_bound":false,"samples":178283},{"bound_ms":100,"rps":{"value":29709,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":8085,"first_disqualified_conc":2048,"lower_bound":false,"samples":178283},{"bound_ms":null,"rps":{"value":29709,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":8085,"first_disqualified_conc":null,"lower_bound":false,"samples":178283}],"sample_counts":{"gateway_c1_samples":32091,"direct_c1_samples":257892},"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:bedrock:gemini","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:gemini:gemini","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:openai:gemini","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":163,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":191,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":219,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":159.98538822669894,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":6250.570824524313,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.98230796385704,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":147826,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":24634,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":16,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.7265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":67.9921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":53.26953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":69.94921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":69.70703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":2.0595721727593674,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.4251936813206195,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":24347,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":507,"first_disqualified_conc":16,"lower_bound":false,"samples":146100},{"bound_ms":5,"rps":{"value":29193,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":4164,"first_disqualified_conc":128,"lower_bound":false,"samples":175189},{"bound_ms":10,"rps":{"value":29862,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7543,"first_disqualified_conc":256,"lower_bound":false,"samples":179198},{"bound_ms":50,"rps":{"value":29862,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7543,"first_disqualified_conc":1024,"lower_bound":false,"samples":179198},{"bound_ms":100,"rps":{"value":29862,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7543,"first_disqualified_conc":2048,"lower_bound":false,"samples":179198},{"bound_ms":null,"rps":{"value":29862,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7543,"first_disqualified_conc":null,"lower_bound":false,"samples":179198}],"sample_counts":{"gateway_c1_samples":31889,"direct_c1_samples":261875},"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:openai-responses:gemini","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":179,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":209,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":237,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":173.738380787614,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5755.7805907173,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9843873210247157,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":136412,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":22732,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":16,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.3359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":67.0234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":52.6640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":68.56640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":68.2265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.08107899161353596,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.8728974207168577,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":24971,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":978,"first_disqualified_conc":32,"lower_bound":false,"samples":149843},{"bound_ms":5,"rps":{"value":27134,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":4762,"first_disqualified_conc":128,"lower_bound":false,"samples":162810},{"bound_ms":10,"rps":{"value":27902,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7820,"first_disqualified_conc":256,"lower_bound":false,"samples":167436},{"bound_ms":50,"rps":{"value":27902,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7820,"first_disqualified_conc":1024,"lower_bound":false,"samples":167436},{"bound_ms":100,"rps":{"value":27902,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7820,"first_disqualified_conc":2048,"lower_bound":false,"samples":167436},{"bound_ms":null,"rps":{"value":27902,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7820,"first_disqualified_conc":null,"lower_bound":false,"samples":167436}],"sample_counts":{"gateway_c1_samples":29315,"direct_c1_samples":264153},"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:cohere:gemini","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:openai-responses:bedrock","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":408,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":458,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":486,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":417.1148158107018,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":2397.4214343271556,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.5155080086482076,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":29752,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4958,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":16,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":63.89453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":57.609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":66.1484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":65.8671875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":2.1686498706373123,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.7747356627383634,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":6806,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":843,"first_disqualified_conc":8,"lower_bound":false,"samples":40846},{"bound_ms":5,"rps":{"value":8726,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":4081,"first_disqualified_conc":32,"lower_bound":false,"samples":52362},{"bound_ms":10,"rps":{"value":9937,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":6708,"first_disqualified_conc":64,"lower_bound":false,"samples":59631},{"bound_ms":50,"rps":{"value":10099,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":12211,"first_disqualified_conc":512,"lower_bound":false,"samples":60598},{"bound_ms":100,"rps":{"value":10099,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":12211,"first_disqualified_conc":1024,"lower_bound":false,"samples":60598},{"bound_ms":null,"rps":{"value":10099,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":12211,"first_disqualified_conc":null,"lower_bound":false,"samples":60598}],"sample_counts":{"gateway_c1_samples":13815,"direct_c1_samples":263299},"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:gemini:bedrock","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:bedrock:bedrock","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:openai:bedrock","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":381,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":427,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":455,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":387.2113676731794,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":2582.5688073394494,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.5886832786469611,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":36595,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":6098,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":16,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.74609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":63.51171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":56.62109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":64.6875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":64.42578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.34857051611065903,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.4981697903891496,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":4228,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":726,"first_disqualified_conc":4,"lower_bound":false,"samples":25376},{"bound_ms":5,"rps":{"value":10244,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2611,"first_disqualified_conc":32,"lower_bound":false,"samples":61474},{"bound_ms":10,"rps":{"value":10358,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":5549,"first_disqualified_conc":64,"lower_bound":false,"samples":62160},{"bound_ms":50,"rps":{"value":10529,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":12445,"first_disqualified_conc":512,"lower_bound":false,"samples":63182},{"bound_ms":100,"rps":{"value":10529,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":12445,"first_disqualified_conc":1024,"lower_bound":false,"samples":63182},{"bound_ms":null,"rps":{"value":10529,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":12445,"first_disqualified_conc":null,"lower_bound":false,"samples":63182}],"sample_counts":{"gateway_c1_samples":14735,"direct_c1_samples":261517},"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:cohere:bedrock","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:anthropic:bedrock","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":400,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":448,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":476,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":389.410061071586,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":2567.9870654810024,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.5138925384814116,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":31766,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":5293,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":16,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.20703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":62.880859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":56.078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":64.55078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":64.265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":1.520035788472686,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.40941641695154907,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":5623,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":788,"first_disqualified_conc":8,"lower_bound":false,"samples":33749},{"bound_ms":5,"rps":{"value":10290,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":4567,"first_disqualified_conc":64,"lower_bound":false,"samples":61750},{"bound_ms":10,"rps":{"value":10340,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":9575,"first_disqualified_conc":128,"lower_bound":false,"samples":62052},{"bound_ms":50,"rps":{"value":10340,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":9575,"first_disqualified_conc":512,"lower_bound":false,"samples":62052},{"bound_ms":100,"rps":{"value":10340,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":9575,"first_disqualified_conc":1024,"lower_bound":false,"samples":62052},{"bound_ms":null,"rps":{"value":10340,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":9575,"first_disqualified_conc":null,"lower_bound":false,"samples":62052}],"sample_counts":{"gateway_c1_samples":14087,"direct_c1_samples":261607},"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:gemini:openai-responses","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:cohere:openai-responses","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:bedrock:openai-responses","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:anthropic:openai-responses","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:openai-responses:openai-responses","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":160,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":188,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":217,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":29,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":148.0885877329253,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":6752.714812862415,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7879501484182734,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":128099,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":21347,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":16,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":34.328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":67.34765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":51.8984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":69.07421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":68.703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.6509917696949409,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.8039072804175704,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":25391,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":479,"first_disqualified_conc":16,"lower_bound":false,"samples":152368},{"bound_ms":5,"rps":{"value":30114,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":3642,"first_disqualified_conc":128,"lower_bound":false,"samples":180699},{"bound_ms":10,"rps":{"value":31044,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7454,"first_disqualified_conc":256,"lower_bound":false,"samples":186270},{"bound_ms":50,"rps":{"value":31044,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7454,"first_disqualified_conc":1024,"lower_bound":false,"samples":186270},{"bound_ms":100,"rps":{"value":31044,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7454,"first_disqualified_conc":2048,"lower_bound":false,"samples":186270},{"bound_ms":null,"rps":{"value":31044,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":7454,"first_disqualified_conc":null,"lower_bound":false,"samples":186270}],"sample_counts":{"gateway_c1_samples":32575,"direct_c1_samples":260507},"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915:openai:openai-responses","gateway_id":"aisix","gateway_build":"ghcr.io/api7/aisix:1.9.0@sha256:dbe230bdaa4f353f7bf99d5437ab11f8b67435afabb217cf371c441ede43270a","run_id":"run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","measured_at":"2026-11-10T17:39:23Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","detail":"Pinned AISIX v1.9.0 source 7e070f0bde3f653c3b197ed18c5b9205b866a4c5: README native adapter families, aisix-proxy/src/lib.rs routes and dispatch.rs::serves_natively_with, provider-vertex/bedrock bridge configuration. OpenAI, Responses and Anthropic ingress translate to Chat Completions, Anthropic, Gemini and Bedrock; only Responses ingress selects the Responses upstream. Separate provider keys use explicit apis:{} for Chat-only routing and the native Responses declaration default for the Responses alias. Thirteen exact pairs passed local recorder checks. Evidence: results/diagnostics/refresh-configuration/aisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.4678530961860305,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89015,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/aisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-aisix-ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"ef0cff28391e0017f17c4b7c01e0171c62359468b787de48402bc3408df76915"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:bedrock:openai","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:openai:openai","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":179,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":340,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":368,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":185.658855916589,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5386.223000583772,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7114382453326605,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":92266,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":15375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":12,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.00045520560119654044,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":10584,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":11228,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (20079us) came in under the mock's (21071us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":8877,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":63882.44005721134,"certified":true,"suppressed":false,"headroom":0.8555683936233661,"rig_ceiling":74666.66666666667,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1470,"certified":true,"suppressed":false,"headroom":0.8555683936233661,"rig_ceiling":74666.66666666667,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":191.2109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":228.22265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":228.3515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":228.44140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":233.2109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":66,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.050196090921940856,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.46382647366702207,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":12187,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":845,"first_disqualified_conc":8,"lower_bound":false,"samples":73136},{"bound_ms":5,"rps":{"value":23089,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":3299,"first_disqualified_conc":64,"lower_bound":false,"samples":138550},{"bound_ms":10,"rps":{"value":25474,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":5650,"first_disqualified_conc":128,"lower_bound":false,"samples":152851},{"bound_ms":50,"rps":{"value":26590,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":19945,"first_disqualified_conc":1024,"lower_bound":false,"samples":159560},{"bound_ms":100,"rps":{"value":26590,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":19945,"first_disqualified_conc":2048,"lower_bound":false,"samples":159560},{"bound_ms":null,"rps":{"value":26590,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":19945,"first_disqualified_conc":32768,"lower_bound":false,"samples":159560}],"sample_counts":{"gateway_c1_samples":28247,"direct_c1_samples":260315,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1280,"http_status_counts":{"200":1280},"stream_errors_no_response":0},{"conc":1408,"http_status_counts":{"200":1408},"stream_errors_no_response":0},{"conc":1472,"http_status_counts":{"200":1472},"stream_errors_no_response":0},{"conc":1440,"http_status_counts":{"200":1440},"stream_errors_no_response":0},{"conc":1456,"http_status_counts":{"200":1456},"stream_errors_no_response":0},{"conc":1464,"http_status_counts":{"200":1464},"stream_errors_no_response":0},{"conc":1468,"http_status_counts":{"200":1468},"stream_errors_no_response":0},{"conc":1470,"http_status_counts":{"200":1470},"stream_errors_no_response":0},{"conc":1471,"http_status_counts":{"200":1471},"stream_errors_no_response":0},{"conc":1470,"http_status_counts":{"200":1470},"stream_errors_no_response":0},{"conc":1470,"http_status_counts":{"200":1470},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:anthropic:openai","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":193,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":373,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":402,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":29,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":198.44375892076062,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5039.211136890952,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7160658692266068,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":86876,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":14477,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":12,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.0005755329435056862,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":10646,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":11159,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (20080us) came in under the mock's (21071us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":9055,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":77198.52773250597,"certified":true,"suppressed":false,"headroom":0.7919989654683227,"rig_ceiling":97473.01587301587,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1919,"certified":true,"suppressed":false,"headroom":0.7919989654683227,"rig_ceiling":97473.01587301587,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":190.64453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":224.5859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":224.65625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":224.83203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":229.5546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":66,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.025964471941633756,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.45247832091501883,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":11298,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":894,"first_disqualified_conc":8,"lower_bound":false,"samples":67801},{"bound_ms":5,"rps":{"value":21698,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":4318,"first_disqualified_conc":64,"lower_bound":false,"samples":130205},{"bound_ms":10,"rps":{"value":24156,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":6860,"first_disqualified_conc":128,"lower_bound":false,"samples":144943},{"bound_ms":50,"rps":{"value":25454,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":20766,"first_disqualified_conc":1024,"lower_bound":false,"samples":152737},{"bound_ms":100,"rps":{"value":25454,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":20766,"first_disqualified_conc":2048,"lower_bound":false,"samples":152737},{"bound_ms":null,"rps":{"value":25454,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":20766,"first_disqualified_conc":16384,"lower_bound":false,"samples":152737}],"sample_counts":{"gateway_c1_samples":26470,"direct_c1_samples":257076,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1792,"http_status_counts":{"200":1792},"stream_errors_no_response":0},{"conc":1920,"http_status_counts":{"200":1920},"stream_errors_no_response":0},{"conc":1856,"http_status_counts":{"200":1856},"stream_errors_no_response":0},{"conc":1888,"http_status_counts":{"200":1888},"stream_errors_no_response":0},{"conc":1904,"http_status_counts":{"200":1904},"stream_errors_no_response":0},{"conc":1912,"http_status_counts":{"200":1912},"stream_errors_no_response":0},{"conc":1916,"http_status_counts":{"200":1916},"stream_errors_no_response":0},{"conc":1918,"http_status_counts":{"200":1918},"stream_errors_no_response":0},{"conc":1919,"http_status_counts":{"200":1919},"stream_errors_no_response":0},{"conc":1919,"http_status_counts":{"200":1919},"stream_errors_no_response":0},{"conc":1919,"http_status_counts":{"200":1919},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:gemini:openai","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:openai-responses:openai","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:cohere:openai","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:gemini:gemini","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:openai-responses:gemini","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:bedrock:gemini","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:openai:gemini","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:cohere:gemini","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:anthropic:gemini","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:anthropic:bedrock","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:openai:bedrock","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:gemini:bedrock","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:bedrock:bedrock","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":234,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":463,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":491,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":244.17659804983748,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":4095.396561286744,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.748941474283438,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":73840,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":12305,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":12,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.0006365113759479957,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":190.5390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":238.904296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":239.44140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":240.44140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":244.83984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.37490060029269184,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.4582365113281113,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":6434,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":778,"first_disqualified_conc":4,"lower_bound":false,"samples":38611},{"bound_ms":5,"rps":{"value":18244,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":4885,"first_disqualified_conc":64,"lower_bound":false,"samples":109475},{"bound_ms":10,"rps":{"value":19877,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":8605,"first_disqualified_conc":128,"lower_bound":false,"samples":119271},{"bound_ms":50,"rps":{"value":20783,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":26573,"first_disqualified_conc":512,"lower_bound":false,"samples":124713},{"bound_ms":100,"rps":{"value":20783,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":26573,"first_disqualified_conc":1024,"lower_bound":false,"samples":124713},{"bound_ms":null,"rps":{"value":20783,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":26573,"first_disqualified_conc":16384,"lower_bound":false,"samples":124713}],"sample_counts":{"gateway_c1_samples":21813,"direct_c1_samples":263299},"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:cohere:bedrock","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:openai-responses:bedrock","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:openai:cohere","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:openai-responses:cohere","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:bedrock:cohere","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:gemini:cohere","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:cohere:cohere","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:anthropic:cohere","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:cohere:anthropic","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:anthropic:anthropic","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":181,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":340,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":368,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":184.98662543312153,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5405.796217205613,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.6807669634417425,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":88601,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":14764,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":12,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.0003724562928183655,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":10516,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":11171,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (20078us) came in under the mock's (21071us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":8998,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":80658.75131817744,"certified":true,"suppressed":false,"headroom":0.8611546456489254,"rig_ceiling":93663.49206349206,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1844,"certified":true,"suppressed":false,"headroom":0.8611546456489254,"rig_ceiling":93663.49206349206,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":193.1328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":230.0234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":230.1953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":230.3984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":235.19140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":66,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.1636566869781283,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.45541914018046065,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":12499,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":797,"first_disqualified_conc":8,"lower_bound":false,"samples":75005},{"bound_ms":5,"rps":{"value":23800,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":3534,"first_disqualified_conc":64,"lower_bound":false,"samples":142817},{"bound_ms":10,"rps":{"value":26599,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":5936,"first_disqualified_conc":128,"lower_bound":false,"samples":159609},{"bound_ms":50,"rps":{"value":27850,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":18633,"first_disqualified_conc":1024,"lower_bound":false,"samples":167116},{"bound_ms":100,"rps":{"value":27850,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":18633,"first_disqualified_conc":2048,"lower_bound":false,"samples":167116},{"bound_ms":null,"rps":{"value":27850,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":18633,"first_disqualified_conc":16384,"lower_bound":false,"samples":167116}],"sample_counts":{"gateway_c1_samples":28060,"direct_c1_samples":258354,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1792,"http_status_counts":{"200":1792},"stream_errors_no_response":0},{"conc":1920,"http_status_counts":{"200":1920},"stream_errors_no_response":0},{"conc":1856,"http_status_counts":{"200":1856},"stream_errors_no_response":0},{"conc":1824,"http_status_counts":{"200":1824},"stream_errors_no_response":0},{"conc":1840,"http_status_counts":{"200":1840},"stream_errors_no_response":0},{"conc":1848,"http_status_counts":{"200":1848},"stream_errors_no_response":0},{"conc":1844,"http_status_counts":{"200":1844},"stream_errors_no_response":0},{"conc":1846,"http_status_counts":{"200":1846},"stream_errors_no_response":0},{"conc":1845,"http_status_counts":{"200":1845},"stream_errors_no_response":0},{"conc":1844,"http_status_counts":{"200":1844},"stream_errors_no_response":0},{"conc":1844,"http_status_counts":{"200":1844},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:openai-responses:anthropic","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:openai:anthropic","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:bedrock:anthropic","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:gemini:anthropic","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:gemini:openai-responses","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:anthropic:openai-responses","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:openai:openai-responses","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:cohere:openai-responses","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:bedrock:openai-responses","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","detail":"Pinned APISIX 3.19.0 source 473043623c161966f3f6ef4e8f12d2d8667a12d2: ai-providers capability tables and ai-protocols/converters registry support four native chat protocol paths plus Anthropic Messages to OpenAI Chat Completions conversion. The Gemini provider uses an OpenAI-compatible endpoint, not native Gemini generateContent; provider brands are not protocol support. Five exact pairs passed local response and recorder checks. Evidence: results/diagnostics/refresh-configuration/apisix-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f:openai-responses:openai-responses","gateway_id":"apisix","gateway_build":"apache/apisix:3.19.0-debian@sha256:7401ce7cfd56de1b9dcecfe7d7c7e5a681bc29222767fd28391995a0936eaa38","run_id":"run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","measured_at":"2026-11-10T17:39:33Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":9.064543290470795,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":104919,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":189,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":358,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":386,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":189.4636194514597,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5278.058145913329,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7572038580695427,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":96219,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":16034,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":12,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.000436504224737318,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":192.4375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":228.27734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":228.3125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":228.4765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":233.046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":66,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.01780973526014571,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.46380000104901226,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":11636,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":885,"first_disqualified_conc":8,"lower_bound":false,"samples":69830},{"bound_ms":5,"rps":{"value":24247,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":3193,"first_disqualified_conc":64,"lower_bound":false,"samples":145490},{"bound_ms":10,"rps":{"value":25825,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":6690,"first_disqualified_conc":128,"lower_bound":false,"samples":154951},{"bound_ms":50,"rps":{"value":28066,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":10749,"first_disqualified_conc":1024,"lower_bound":false,"samples":168422},{"bound_ms":100,"rps":{"value":28066,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":10749,"first_disqualified_conc":2048,"lower_bound":false,"samples":168422},{"bound_ms":null,"rps":{"value":28066,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":10749,"first_disqualified_conc":32768,"lower_bound":false,"samples":168422}],"sample_counts":{"gateway_c1_samples":26997,"direct_c1_samples":261509},"provenance":{"source_url":"https://onthebench.ai/gateways/apisix","raw_result_url":"https://onthebench.ai/api/v1/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-apisix-cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"cc145fc7e19bd52b5c47dc4c3db32fb0511f656418b54e198d2b3cf922523d3f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:openai:openai","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":658,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1057,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1084,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":865.1196519216824,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1155.9094719195307,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9909277682234482,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":27580,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4595,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.02708484408992023,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":932,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":1417,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":3,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21148us) came in under the mock's (21189us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":25840.89500725109,"certified":true,"suppressed":false,"headroom":0.8771424490607859,"rig_ceiling":29460.31746031746,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":580,"certified":true,"suppressed":false,"headroom":0.8771424490607859,"rig_ceiling":29460.31746031746,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":160.2734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1049.515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1256.6328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1257.0390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1137.43359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":56.82213581346736,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.4802766176196459,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4346,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2689,"first_disqualified_conc":8,"lower_bound":false,"samples":26081},{"bound_ms":10,"rps":{"value":4555,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5837,"first_disqualified_conc":16,"lower_bound":false,"samples":27340},{"bound_ms":50,"rps":{"value":4624,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":11451,"first_disqualified_conc":128,"lower_bound":false,"samples":27748},{"bound_ms":100,"rps":{"value":4624,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":11451,"first_disqualified_conc":256,"lower_bound":false,"samples":27748},{"bound_ms":null,"rps":{"value":4624,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":11451,"first_disqualified_conc":null,"lower_bound":false,"samples":27748}],"sample_counts":{"gateway_c1_samples":8543,"direct_c1_samples":269310,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":768,"http_status_counts":{"200":768},"stream_errors_no_response":0},{"conc":640,"http_status_counts":{"200":640},"stream_errors_no_response":0},{"conc":576,"http_status_counts":{"200":576},"stream_errors_no_response":0},{"conc":608,"http_status_counts":{"200":608},"stream_errors_no_response":0},{"conc":592,"http_status_counts":{"200":592},"stream_errors_no_response":0},{"conc":584,"http_status_counts":{"200":584},"stream_errors_no_response":0},{"conc":580,"http_status_counts":{"200":580},"stream_errors_no_response":0},{"conc":582,"http_status_counts":{"200":582},"stream_errors_no_response":0},{"conc":581,"http_status_counts":{"200":581},"stream_errors_no_response":0},{"conc":580,"http_status_counts":{"200":580},"stream_errors_no_response":0},{"conc":580,"http_status_counts":{"200":580},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:openai-responses:openai","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":586,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":944,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":971,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":842.668921299232,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1186.705685618729,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9933326877124895,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":28386,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4730,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.00542520961037131,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":161.203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1095.1640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1073.67578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1252.703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1167.71484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":73.1085228333916,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.35490141220737104,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4373,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2805,"first_disqualified_conc":8,"lower_bound":false,"samples":26247},{"bound_ms":10,"rps":{"value":4576,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5339,"first_disqualified_conc":16,"lower_bound":false,"samples":27465},{"bound_ms":50,"rps":{"value":4629,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":10669,"first_disqualified_conc":128,"lower_bound":false,"samples":27782},{"bound_ms":100,"rps":{"value":4629,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":10669,"first_disqualified_conc":256,"lower_bound":false,"samples":27782},{"bound_ms":null,"rps":{"value":4629,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":10669,"first_disqualified_conc":null,"lower_bound":false,"samples":27782}],"sample_counts":{"gateway_c1_samples":9572,"direct_c1_samples":263867},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:anthropic:openai","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":597,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":968,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":996,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":863.6478806964202,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1157.8792958927074,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9909830628870169,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":27627,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4603,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":868,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":1061,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":4,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":29329.081029339704,"certified":true,"suppressed":false,"headroom":0.8516464347568222,"rig_ceiling":34438.09523809524,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":678,"certified":true,"suppressed":false,"headroom":0.8516464347568222,"rig_ceiling":34438.09523809524,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":161.00390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1099.4375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1073.2734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1300.45703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1209.50390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":88.44650309718233,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.9297672905162758,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":1531,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":994,"first_disqualified_conc":2,"lower_bound":false,"samples":9192},{"bound_ms":5,"rps":{"value":4343,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2710,"first_disqualified_conc":8,"lower_bound":false,"samples":26062},{"bound_ms":10,"rps":{"value":4579,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5802,"first_disqualified_conc":16,"lower_bound":false,"samples":27481},{"bound_ms":50,"rps":{"value":4623,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":11928,"first_disqualified_conc":128,"lower_bound":false,"samples":27743},{"bound_ms":100,"rps":{"value":4623,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":11928,"first_disqualified_conc":256,"lower_bound":false,"samples":27743},{"bound_ms":null,"rps":{"value":4623,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":11928,"first_disqualified_conc":null,"lower_bound":false,"samples":27743}],"sample_counts":{"gateway_c1_samples":9413,"direct_c1_samples":262256,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":768,"http_status_counts":{"200":768},"stream_errors_no_response":0},{"conc":640,"http_status_counts":{"200":640},"stream_errors_no_response":0},{"conc":704,"http_status_counts":{"200":704},"stream_errors_no_response":0},{"conc":672,"http_status_counts":{"200":672},"stream_errors_no_response":0},{"conc":688,"http_status_counts":{"200":688},"stream_errors_no_response":0},{"conc":680,"http_status_counts":{"200":680},"stream_errors_no_response":0},{"conc":676,"http_status_counts":{"200":676},"stream_errors_no_response":0},{"conc":678,"http_status_counts":{"200":678},"stream_errors_no_response":0},{"conc":679,"http_status_counts":{"200":679},"stream_errors_no_response":0},{"conc":678,"http_status_counts":{"200":678},"stream_errors_no_response":0},{"conc":678,"http_status_counts":{"200":678},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:gemini:openai","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":588,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":972,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1000,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":846.2167253083129,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1181.7303653926922,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9890099259774329,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":28137,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4688,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.021039911859828696,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":162.01171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1091.18359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1074.71875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1249.7890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1190.69921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":81.07866820710633,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.2632439098184473,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":1547,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":990,"first_disqualified_conc":2,"lower_bound":false,"samples":9287},{"bound_ms":5,"rps":{"value":4450,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2567,"first_disqualified_conc":8,"lower_bound":false,"samples":26704},{"bound_ms":10,"rps":{"value":4646,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5555,"first_disqualified_conc":16,"lower_bound":false,"samples":27884},{"bound_ms":50,"rps":{"value":4724,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12204,"first_disqualified_conc":128,"lower_bound":false,"samples":28353},{"bound_ms":100,"rps":{"value":4724,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12204,"first_disqualified_conc":256,"lower_bound":false,"samples":28353},{"bound_ms":null,"rps":{"value":4724,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12204,"first_disqualified_conc":null,"lower_bound":false,"samples":28353}],"sample_counts":{"gateway_c1_samples":9465,"direct_c1_samples":259786},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:cohere:openai","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":572,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":912,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":946,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":34,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":813.5755585206473,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1229.1421362489486,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9878134274950701,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":29229,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4870,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.02292244004242362,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":158.37890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1049.91796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1259.31640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1259.328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1138.40234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":78.12778798490757,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.9327641164400249,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":1624,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":999,"first_disqualified_conc":2,"lower_bound":false,"samples":9748},{"bound_ms":5,"rps":{"value":4853,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4977,"first_disqualified_conc":16,"lower_bound":false,"samples":29122},{"bound_ms":10,"rps":{"value":4853,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4977,"first_disqualified_conc":16,"lower_bound":false,"samples":29122},{"bound_ms":50,"rps":{"value":4898,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":10790,"first_disqualified_conc":128,"lower_bound":false,"samples":29394},{"bound_ms":100,"rps":{"value":4898,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":10790,"first_disqualified_conc":256,"lower_bound":false,"samples":29394},{"bound_ms":null,"rps":{"value":4898,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":10790,"first_disqualified_conc":null,"lower_bound":false,"samples":29394}],"sample_counts":{"gateway_c1_samples":9654,"direct_c1_samples":215609},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:bedrock:openai","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":632,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1148,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1176,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":909.2650044046114,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1099.7893850042124,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9856819939844713,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":26109,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4350,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.0371902409130951,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":160.41796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1042.69921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1232.00390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1234.2578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1146.58203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":73.76935084154414,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.1681216604557927,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4096,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2742,"first_disqualified_conc":8,"lower_bound":false,"samples":24582},{"bound_ms":10,"rps":{"value":4328,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5817,"first_disqualified_conc":16,"lower_bound":false,"samples":25973},{"bound_ms":50,"rps":{"value":4389,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12493,"first_disqualified_conc":128,"lower_bound":false,"samples":26340},{"bound_ms":100,"rps":{"value":4389,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12493,"first_disqualified_conc":256,"lower_bound":false,"samples":26340},{"bound_ms":null,"rps":{"value":4389,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12493,"first_disqualified_conc":null,"lower_bound":false,"samples":26340}],"sample_counts":{"gateway_c1_samples":8716,"direct_c1_samples":262374},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:openai:openai-responses","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Bifrost v2.5.0 source 69f4d56d18b9ba00ac5f9cf969b4501a882b9967. Every upstream except Bedrock uses the field's common model gpt-4o-mini; Bedrock uses a Claude model id (anthropic.claude-3-5-sonnet-20240620-v1:0) so Bifrost stays on the bedrock-runtime Converse surface rather than routing an OpenAI model family to its OpenAI-compatible Mantle surface (core/providers/bedrock/bedrock.go resolveSurface). OpenAI Chat and Cohere ingress into the Responses-only upstream are not enabled: Bifrost only converts them when the model catalog marks the model Responses-only (plugins/compat/main.go::markForConversion), and selecting a different model (o3-pro) to trigger that path also triggers the compat plugin's default parameter dropping (it removed max_completion_tokens in the preflight), which the benchmark's out-of-the-box rule excludes. The Bedrock upstream uses Bifrost's default HTTPS runtime host (bedrock-runtime.us-east-1.amazonaws.com), which the engine v1.2 catch-all DNS resolves to the mock; its six cells are probed and carry upstream_transport \"tls\".","detail":"Bifrost v2.5.0 source 69f4d56d18b9ba00ac5f9cf969b4501a882b9967. Every upstream except Bedrock uses the field's common model gpt-4o-mini; Bedrock uses a Claude model id (anthropic.claude-3-5-sonnet-20240620-v1:0) so Bifrost stays on the bedrock-runtime Converse surface rather than routing an OpenAI model family to its OpenAI-compatible Mantle surface (core/providers/bedrock/bedrock.go resolveSurface). OpenAI Chat and Cohere ingress into the Responses-only upstream are not enabled: Bifrost only converts them when the model catalog marks the model Responses-only (plugins/compat/main.go::markForConversion), and selecting a different model (o3-pro) to trigger that path also triggers the compat plugin's default parameter dropping (it removed max_completion_tokens in the preflight), which the benchmark's out-of-the-box rule excludes. The Bedrock upstream uses Bifrost's default HTTPS runtime host (bedrock-runtime.us-east-1.amazonaws.com), which the engine v1.2 catch-all DNS resolves to the mock; its six cells are probed and carry upstream_transport \"tls\".","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:openai-responses:openai-responses","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":631,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1001,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1029,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":882.23316426641,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1133.487201007134,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9899135051863024,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":27011,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4501,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":15,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.019510569767872347,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":160.6015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1093.65625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1288.046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1288.0703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1178.140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":64.4174808472209,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.22443919578251986,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4320,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2453,"first_disqualified_conc":8,"lower_bound":false,"samples":25924},{"bound_ms":10,"rps":{"value":4443,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":6180,"first_disqualified_conc":16,"lower_bound":false,"samples":26666},{"bound_ms":50,"rps":{"value":4471,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":11655,"first_disqualified_conc":128,"lower_bound":false,"samples":26831},{"bound_ms":100,"rps":{"value":4471,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":11655,"first_disqualified_conc":256,"lower_bound":false,"samples":26831},{"bound_ms":null,"rps":{"value":4471,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":11655,"first_disqualified_conc":null,"lower_bound":false,"samples":26831}],"sample_counts":{"gateway_c1_samples":8878,"direct_c1_samples":266170},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:anthropic:openai-responses","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":641,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1011,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1038,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":886.3399374348279,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1128.235294117647,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9886562313460872,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":26852,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4474,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.02267987486965589,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":164.2421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1105.4921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1137.90234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1237.328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1209.71484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":83.21399179381376,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.9698354595007855,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4161,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2717,"first_disqualified_conc":8,"lower_bound":false,"samples":24972},{"bound_ms":10,"rps":{"value":4330,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5739,"first_disqualified_conc":16,"lower_bound":false,"samples":25985},{"bound_ms":50,"rps":{"value":4391,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":11784,"first_disqualified_conc":128,"lower_bound":false,"samples":26357},{"bound_ms":100,"rps":{"value":4391,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":11784,"first_disqualified_conc":256,"lower_bound":false,"samples":26357},{"bound_ms":null,"rps":{"value":4391,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":11784,"first_disqualified_conc":null,"lower_bound":false,"samples":26357}],"sample_counts":{"gateway_c1_samples":8756,"direct_c1_samples":262163},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:gemini:openai-responses","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":653,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1008,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1035,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":901.3135480940304,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1109.4918101637968,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9888138381760533,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":26417,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4402,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":159.6796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1083.3671875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1239.97265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1240.00390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1183.16015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":66.32112791437604,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.5692884595172528,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4182,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2716,"first_disqualified_conc":8,"lower_bound":false,"samples":25101},{"bound_ms":10,"rps":{"value":4323,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5487,"first_disqualified_conc":16,"lower_bound":false,"samples":25944},{"bound_ms":50,"rps":{"value":4375,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12311,"first_disqualified_conc":128,"lower_bound":false,"samples":26254},{"bound_ms":100,"rps":{"value":4375,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12311,"first_disqualified_conc":256,"lower_bound":false,"samples":26254},{"bound_ms":null,"rps":{"value":4375,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12311,"first_disqualified_conc":null,"lower_bound":false,"samples":26254}],"sample_counts":{"gateway_c1_samples":8613,"direct_c1_samples":269135},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:cohere:openai-responses","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Bifrost v2.5.0 source 69f4d56d18b9ba00ac5f9cf969b4501a882b9967. Every upstream except Bedrock uses the field's common model gpt-4o-mini; Bedrock uses a Claude model id (anthropic.claude-3-5-sonnet-20240620-v1:0) so Bifrost stays on the bedrock-runtime Converse surface rather than routing an OpenAI model family to its OpenAI-compatible Mantle surface (core/providers/bedrock/bedrock.go resolveSurface). OpenAI Chat and Cohere ingress into the Responses-only upstream are not enabled: Bifrost only converts them when the model catalog marks the model Responses-only (plugins/compat/main.go::markForConversion), and selecting a different model (o3-pro) to trigger that path also triggers the compat plugin's default parameter dropping (it removed max_completion_tokens in the preflight), which the benchmark's out-of-the-box rule excludes. The Bedrock upstream uses Bifrost's default HTTPS runtime host (bedrock-runtime.us-east-1.amazonaws.com), which the engine v1.2 catch-all DNS resolves to the mock; its six cells are probed and carry upstream_transport \"tls\".","detail":"Bifrost v2.5.0 source 69f4d56d18b9ba00ac5f9cf969b4501a882b9967. Every upstream except Bedrock uses the field's common model gpt-4o-mini; Bedrock uses a Claude model id (anthropic.claude-3-5-sonnet-20240620-v1:0) so Bifrost stays on the bedrock-runtime Converse surface rather than routing an OpenAI model family to its OpenAI-compatible Mantle surface (core/providers/bedrock/bedrock.go resolveSurface). OpenAI Chat and Cohere ingress into the Responses-only upstream are not enabled: Bifrost only converts them when the model catalog marks the model Responses-only (plugins/compat/main.go::markForConversion), and selecting a different model (o3-pro) to trigger that path also triggers the compat plugin's default parameter dropping (it removed max_completion_tokens in the preflight), which the benchmark's out-of-the-box rule excludes. The Bedrock upstream uses Bifrost's default HTTPS runtime host (bedrock-runtime.us-east-1.amazonaws.com), which the engine v1.2 catch-all DNS resolves to the mock; its six cells are probed and carry upstream_transport \"tls\".","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:bedrock:openai-responses","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":611,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":958,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":985,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":851.7911829641763,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1173.9966555183946,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9932927434109152,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":28082,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4679,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.04376468912470622,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":161.5546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1083.189453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1295.55859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1295.55859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1184.015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":85.1344155013595,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.5264685233519769,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4197,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2793,"first_disqualified_conc":8,"lower_bound":false,"samples":25188},{"bound_ms":10,"rps":{"value":4442,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5989,"first_disqualified_conc":16,"lower_bound":false,"samples":26658},{"bound_ms":50,"rps":{"value":4547,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":24229,"first_disqualified_conc":128,"lower_bound":false,"samples":27285},{"bound_ms":100,"rps":{"value":4547,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":24229,"first_disqualified_conc":256,"lower_bound":false,"samples":27285},{"bound_ms":null,"rps":{"value":4547,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":24229,"first_disqualified_conc":null,"lower_bound":false,"samples":27285}],"sample_counts":{"gateway_c1_samples":9156,"direct_c1_samples":264307},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:openai:anthropic","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":673,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1094,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1122,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":931.3495782567948,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1073.7106918238992,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9905722786350731,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":25608,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4267,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.021829115901280848,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":857,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":1039,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":7,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":25,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":26538.00329954379,"certified":true,"suppressed":false,"headroom":0.8372867627560391,"rig_ceiling":31695.238095238095,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":624,"certified":true,"suppressed":false,"headroom":0.8372867627560391,"rig_ceiling":31695.238095238095,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":162.32421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1067.142578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1319.26171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1319.3046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1199.890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":63.28789981255695,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.6335511770764554,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4001,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":3018,"first_disqualified_conc":8,"lower_bound":false,"samples":24014},{"bound_ms":10,"rps":{"value":4117,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":6003,"first_disqualified_conc":16,"lower_bound":false,"samples":24710},{"bound_ms":50,"rps":{"value":4177,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":28052,"first_disqualified_conc":128,"lower_bound":false,"samples":25067},{"bound_ms":100,"rps":{"value":4177,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":28052,"first_disqualified_conc":256,"lower_bound":false,"samples":25067},{"bound_ms":null,"rps":{"value":4177,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":28052,"first_disqualified_conc":null,"lower_bound":false,"samples":25067}],"sample_counts":{"gateway_c1_samples":8297,"direct_c1_samples":262017,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":768,"http_status_counts":{"200":768},"stream_errors_no_response":0},{"conc":640,"http_status_counts":{"200":640},"stream_errors_no_response":0},{"conc":576,"http_status_counts":{"200":576},"stream_errors_no_response":0},{"conc":608,"http_status_counts":{"200":608},"stream_errors_no_response":0},{"conc":624,"http_status_counts":{"200":624},"stream_errors_no_response":0},{"conc":632,"http_status_counts":{"200":632},"stream_errors_no_response":0},{"conc":628,"http_status_counts":{"200":628},"stream_errors_no_response":0},{"conc":626,"http_status_counts":{"200":626},"stream_errors_no_response":0},{"conc":625,"http_status_counts":{"200":625},"stream_errors_no_response":0},{"conc":624,"http_status_counts":{"200":624},"stream_errors_no_response":0},{"conc":624,"http_status_counts":{"200":624},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:openai-responses:anthropic","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":770,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1259,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1287,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1057.1097647897363,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":945.9755583649389,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9855767917888869,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":22448,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":3740,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":159.08984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":984.9296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1050.9609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1220.609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1174.5546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":168.51907253239412,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.4696816087972396,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":3458,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":3364,"first_disqualified_conc":8,"lower_bound":false,"samples":20760},{"bound_ms":10,"rps":{"value":3703,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":7265,"first_disqualified_conc":16,"lower_bound":false,"samples":22224},{"bound_ms":50,"rps":{"value":3703,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":7265,"first_disqualified_conc":128,"lower_bound":false,"samples":22224},{"bound_ms":100,"rps":{"value":3703,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":7265,"first_disqualified_conc":256,"lower_bound":false,"samples":22224},{"bound_ms":null,"rps":{"value":3703,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":7265,"first_disqualified_conc":null,"lower_bound":false,"samples":22224}],"sample_counts":{"gateway_c1_samples":7332,"direct_c1_samples":263386},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:anthropic:anthropic","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":678,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1101,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1129,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1001.9235594212595,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":998.0801335559265,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9949352467472666,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":23914,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":3985,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.036882161077193276,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":826,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":924,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21071us) came in under the mock's (21073us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21138us) came in under the mock's (21219us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":28037.49790079561,"certified":true,"suppressed":false,"headroom":0.8081819032531677,"rig_ceiling":34692.06349206349,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":683,"certified":true,"suppressed":false,"headroom":0.8081819032531677,"rig_ceiling":34692.06349206349,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":161.85546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1059.20703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1112.515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1274.21875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1215.40625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":80.73395904462717,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.7146363497807083,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":3861,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":3069,"first_disqualified_conc":8,"lower_bound":false,"samples":23174},{"bound_ms":10,"rps":{"value":3988,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":6633,"first_disqualified_conc":16,"lower_bound":false,"samples":23931},{"bound_ms":50,"rps":{"value":4035,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":13944,"first_disqualified_conc":128,"lower_bound":false,"samples":24216},{"bound_ms":100,"rps":{"value":4035,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":13944,"first_disqualified_conc":256,"lower_bound":false,"samples":24216},{"bound_ms":null,"rps":{"value":4035,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":13944,"first_disqualified_conc":null,"lower_bound":false,"samples":24216}],"sample_counts":{"gateway_c1_samples":8283,"direct_c1_samples":258757,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":768,"http_status_counts":{"200":768},"stream_errors_no_response":0},{"conc":896,"http_status_counts":{"200":896},"stream_errors_no_response":0},{"conc":832,"http_status_counts":{"200":832},"stream_errors_no_response":0},{"conc":864,"http_status_counts":{"200":864},"stream_errors_no_response":0},{"conc":848,"http_status_counts":{"200":848},"stream_errors_no_response":0},{"conc":856,"http_status_counts":{"200":856},"stream_errors_no_response":0},{"conc":852,"http_status_counts":{"200":852},"stream_errors_no_response":0},{"conc":854,"http_status_counts":{"200":854},"stream_errors_no_response":0},{"conc":855,"http_status_counts":{"200":855},"stream_errors_no_response":0},{"conc":855,"http_status_counts":{"200":855},"stream_errors_no_response":0},{"conc":855,"http_status_counts":{"200":855},"stream_errors_no_response":0},{"conc":683,"http_status_counts":{"200":683},"stream_errors_no_response":0},{"conc":683,"http_status_counts":{"200":683},"stream_errors_no_response":0},{"conc":683,"http_status_counts":{"200":683},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:gemini:anthropic","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":693,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1153,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1181,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":998.9980796526676,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1001.0029251984956,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9937373062060388,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":23954,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":3991,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":16,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":161.1328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1047.7734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1109.0703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1249.7578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1215.25,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":96.97735724619463,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.21219393604779083,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":3872,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":3197,"first_disqualified_conc":8,"lower_bound":false,"samples":23239},{"bound_ms":10,"rps":{"value":3995,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":6262,"first_disqualified_conc":16,"lower_bound":false,"samples":23976},{"bound_ms":50,"rps":{"value":4040,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12578,"first_disqualified_conc":128,"lower_bound":false,"samples":24249},{"bound_ms":100,"rps":{"value":4040,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12578,"first_disqualified_conc":256,"lower_bound":false,"samples":24249},{"bound_ms":null,"rps":{"value":4040,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12578,"first_disqualified_conc":null,"lower_bound":false,"samples":24249}],"sample_counts":{"gateway_c1_samples":8044,"direct_c1_samples":259231},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:cohere:anthropic","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":685,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1201,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1230,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":29,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":946.1870960066622,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1056.8734283319363,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9910715187644553,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":25217,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4202,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.019708926517825277,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":163.01953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1019.55859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":961.4296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1261.3125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1158.99609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":69.14945491716075,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.26834700549162155,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":3987,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":3067,"first_disqualified_conc":8,"lower_bound":false,"samples":23927},{"bound_ms":10,"rps":{"value":4149,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":6056,"first_disqualified_conc":16,"lower_bound":false,"samples":24902},{"bound_ms":50,"rps":{"value":4224,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":13071,"first_disqualified_conc":128,"lower_bound":false,"samples":25349},{"bound_ms":100,"rps":{"value":4224,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":13071,"first_disqualified_conc":256,"lower_bound":false,"samples":25349},{"bound_ms":null,"rps":{"value":4224,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":13071,"first_disqualified_conc":null,"lower_bound":false,"samples":25349}],"sample_counts":{"gateway_c1_samples":8100,"direct_c1_samples":245712},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:bedrock:anthropic","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":620,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1030,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1057,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":869.4860306706007,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1150.1047339757017,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9910610498359808,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":27453,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4574,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":160.27734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1059.34375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1283.984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1283.98828125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1155.984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":44.08274820353458,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.5597976599390831,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4524,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4996,"first_disqualified_conc":16,"lower_bound":false,"samples":27150},{"bound_ms":10,"rps":{"value":4524,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4996,"first_disqualified_conc":16,"lower_bound":false,"samples":27150},{"bound_ms":50,"rps":{"value":4633,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12131,"first_disqualified_conc":128,"lower_bound":false,"samples":27807},{"bound_ms":100,"rps":{"value":4633,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12131,"first_disqualified_conc":256,"lower_bound":false,"samples":27807},{"bound_ms":null,"rps":{"value":4633,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12131,"first_disqualified_conc":null,"lower_bound":false,"samples":27807}],"sample_counts":{"gateway_c1_samples":8987,"direct_c1_samples":265983},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:openai:gemini","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":626,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1018,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1046,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":913.8050241991242,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1094.3253467843633,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9882617364917406,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":26034,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4338,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":162.02734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1093.5859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1193.640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1199.39453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1200.12890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":73.29932566996371,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.380561021774131,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4251,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2743,"first_disqualified_conc":8,"lower_bound":false,"samples":25514},{"bound_ms":10,"rps":{"value":4415,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5607,"first_disqualified_conc":16,"lower_bound":false,"samples":26497},{"bound_ms":50,"rps":{"value":4467,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12110,"first_disqualified_conc":128,"lower_bound":false,"samples":26809},{"bound_ms":100,"rps":{"value":4467,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12110,"first_disqualified_conc":256,"lower_bound":false,"samples":26809},{"bound_ms":null,"rps":{"value":4467,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12110,"first_disqualified_conc":null,"lower_bound":false,"samples":26809}],"sample_counts":{"gateway_c1_samples":8969,"direct_c1_samples":262364},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:openai-responses:gemini","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":667,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1139,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1169,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":958.5794062110803,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1043.2103939647945,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9910643330296622,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":24891,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4147,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":161.02734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1065.328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1112.6953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1287.9140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1203.3515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":57.62264253641225,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.16256131294155626,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4033,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2855,"first_disqualified_conc":8,"lower_bound":false,"samples":24211},{"bound_ms":10,"rps":{"value":4174,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5878,"first_disqualified_conc":16,"lower_bound":false,"samples":25048},{"bound_ms":50,"rps":{"value":4222,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12585,"first_disqualified_conc":128,"lower_bound":false,"samples":25335},{"bound_ms":100,"rps":{"value":4222,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12585,"first_disqualified_conc":256,"lower_bound":false,"samples":25335},{"bound_ms":null,"rps":{"value":4222,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12585,"first_disqualified_conc":null,"lower_bound":false,"samples":25335}],"sample_counts":{"gateway_c1_samples":8355,"direct_c1_samples":244434},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:anthropic:gemini","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":634,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1153,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":926.2870514820593,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1079.578947368421,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.986098460060681,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":25640,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4272,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.014430577223088924,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":160.82421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1083.51953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1179.125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1239.10546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1228.04296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":81.52955434849295,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.7108664636303431,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4055,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2808,"first_disqualified_conc":8,"lower_bound":false,"samples":24334},{"bound_ms":10,"rps":{"value":4182,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":6177,"first_disqualified_conc":16,"lower_bound":false,"samples":25101},{"bound_ms":50,"rps":{"value":4257,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12666,"first_disqualified_conc":128,"lower_bound":false,"samples":25547},{"bound_ms":100,"rps":{"value":4257,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12666,"first_disqualified_conc":256,"lower_bound":false,"samples":25547},{"bound_ms":null,"rps":{"value":4257,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12666,"first_disqualified_conc":null,"lower_bound":false,"samples":25547}],"sample_counts":{"gateway_c1_samples":8607,"direct_c1_samples":257441},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:gemini:gemini","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":652,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1098,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":954.9274255156608,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1047.2,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9866296894664301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":24871,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4144,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":161.47265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1066.12109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":863.8125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1200.625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1200.6953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":86.4146376641646,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.6505053739690356,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":3959,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":3181,"first_disqualified_conc":8,"lower_bound":false,"samples":23762},{"bound_ms":10,"rps":{"value":4109,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5919,"first_disqualified_conc":16,"lower_bound":false,"samples":24661},{"bound_ms":50,"rps":{"value":4162,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":13496,"first_disqualified_conc":128,"lower_bound":false,"samples":24977},{"bound_ms":100,"rps":{"value":4162,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":13496,"first_disqualified_conc":256,"lower_bound":false,"samples":24977},{"bound_ms":null,"rps":{"value":4162,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":13496,"first_disqualified_conc":null,"lower_bound":false,"samples":24977}],"sample_counts":{"gateway_c1_samples":8589,"direct_c1_samples":266124},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:cohere:gemini","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":658,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1090,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1120,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":953.6970279580229,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1048.5510289794204,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9889482427961908,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":24966,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4160,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":15,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":160.05859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":986.193359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1046.546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1179.5234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1142.328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":69.0800380979254,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.438873930132072,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4011,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2514,"first_disqualified_conc":8,"lower_bound":false,"samples":24072},{"bound_ms":10,"rps":{"value":4196,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5990,"first_disqualified_conc":16,"lower_bound":false,"samples":25181},{"bound_ms":50,"rps":{"value":4233,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":13246,"first_disqualified_conc":128,"lower_bound":false,"samples":25400},{"bound_ms":100,"rps":{"value":4233,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":13246,"first_disqualified_conc":256,"lower_bound":false,"samples":25400},{"bound_ms":null,"rps":{"value":4233,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":13246,"first_disqualified_conc":null,"lower_bound":false,"samples":25400}],"sample_counts":{"gateway_c1_samples":8497,"direct_c1_samples":243412},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:bedrock:gemini","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":607,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1009,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1038,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":29,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":898.0235966032915,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1113.5564853556486,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9925984018721842,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":26614,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4435,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":160.74609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1057.921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":928.47265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1270.5859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1157.69921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":64.94421763716046,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.33456776025435664,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":1551,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":980,"first_disqualified_conc":2,"lower_bound":false,"samples":9310},{"bound_ms":5,"rps":{"value":4311,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2846,"first_disqualified_conc":8,"lower_bound":false,"samples":25871},{"bound_ms":10,"rps":{"value":4413,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":6049,"first_disqualified_conc":16,"lower_bound":false,"samples":26482},{"bound_ms":50,"rps":{"value":4603,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12014,"first_disqualified_conc":128,"lower_bound":false,"samples":27624},{"bound_ms":100,"rps":{"value":4603,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12014,"first_disqualified_conc":256,"lower_bound":false,"samples":27624},{"bound_ms":null,"rps":{"value":4603,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12014,"first_disqualified_conc":null,"lower_bound":false,"samples":27624}],"sample_counts":{"gateway_c1_samples":9211,"direct_c1_samples":248434},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:openai:cohere","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":641,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1090,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1118,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":900.0679296550683,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1111.027253668763,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9906532496425215,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":26498,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4415,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":161.38671875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1119.109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1071.81640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1254.93359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1211.171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":74.02020919745377,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.15608876339144756,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4074,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2833,"first_disqualified_conc":8,"lower_bound":false,"samples":24453},{"bound_ms":10,"rps":{"value":4339,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5792,"first_disqualified_conc":16,"lower_bound":false,"samples":26039},{"bound_ms":50,"rps":{"value":4507,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":24297,"first_disqualified_conc":128,"lower_bound":false,"samples":27048},{"bound_ms":100,"rps":{"value":4507,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":24297,"first_disqualified_conc":256,"lower_bound":false,"samples":27048},{"bound_ms":null,"rps":{"value":4507,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":24297,"first_disqualified_conc":null,"lower_bound":false,"samples":27048}],"sample_counts":{"gateway_c1_samples":8677,"direct_c1_samples":263654},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:openai-responses:cohere","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":630,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1032,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1061,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":29,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":907.9445145018916,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1101.388888888889,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9869553319558813,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":26169,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4360,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.03695211891933203,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":161.4140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1074.9375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1037.11328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1256.90234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1184.60546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":79.7896650787927,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.5104323063992409,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4135,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2944,"first_disqualified_conc":8,"lower_bound":false,"samples":24816},{"bound_ms":10,"rps":{"value":4290,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5892,"first_disqualified_conc":16,"lower_bound":false,"samples":25751},{"bound_ms":50,"rps":{"value":4330,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":11967,"first_disqualified_conc":128,"lower_bound":false,"samples":25988},{"bound_ms":100,"rps":{"value":4330,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":11967,"first_disqualified_conc":256,"lower_bound":false,"samples":25988},{"bound_ms":null,"rps":{"value":4330,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":11967,"first_disqualified_conc":null,"lower_bound":false,"samples":25988}],"sample_counts":{"gateway_c1_samples":8900,"direct_c1_samples":251031},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:anthropic:cohere","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":608,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1061,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1089,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":892.6769092744197,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1120.2261306532664,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9914790593990157,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":26751,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4457,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.0182796904788606,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":161.25390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1118.10546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1140.0546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1222.47265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1216.37109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":68.59097881341208,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.16652196535801708,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4182,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":3030,"first_disqualified_conc":8,"lower_bound":false,"samples":25097},{"bound_ms":10,"rps":{"value":4364,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5657,"first_disqualified_conc":16,"lower_bound":false,"samples":26188},{"bound_ms":50,"rps":{"value":4404,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12287,"first_disqualified_conc":128,"lower_bound":false,"samples":26429},{"bound_ms":100,"rps":{"value":4404,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12287,"first_disqualified_conc":256,"lower_bound":false,"samples":26429},{"bound_ms":null,"rps":{"value":4404,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12287,"first_disqualified_conc":null,"lower_bound":false,"samples":26429}],"sample_counts":{"gateway_c1_samples":9106,"direct_c1_samples":260492},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:gemini:cohere","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":670,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1111,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1138,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":944.4290190789735,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1058.8408231835365,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9888520783721307,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":25211,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4200,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.04561500932132799,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":162.3125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1062.96484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1089.31640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1247.7734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1208.5546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":89.15210308843142,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.28888128372515204,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4019,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":3181,"first_disqualified_conc":8,"lower_bound":false,"samples":24118},{"bound_ms":10,"rps":{"value":4163,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5970,"first_disqualified_conc":16,"lower_bound":false,"samples":24987},{"bound_ms":50,"rps":{"value":4191,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":13106,"first_disqualified_conc":128,"lower_bound":false,"samples":25155},{"bound_ms":100,"rps":{"value":4191,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":13106,"first_disqualified_conc":256,"lower_bound":false,"samples":25155},{"bound_ms":null,"rps":{"value":4191,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":13106,"first_disqualified_conc":null,"lower_bound":false,"samples":25155}],"sample_counts":{"gateway_c1_samples":8377,"direct_c1_samples":268876},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:cohere:cohere","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":609,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1062,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1090,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":888.3598080428555,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1125.6700167504189,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9919159700788478,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":26881,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4479,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.01860049849335962,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":160.7265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1074.453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1027.63671875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1245.73046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1158.9921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":76.39987441289486,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.6278166214623293,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4379,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2599,"first_disqualified_conc":8,"lower_bound":false,"samples":26284},{"bound_ms":10,"rps":{"value":4498,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5640,"first_disqualified_conc":16,"lower_bound":false,"samples":26992},{"bound_ms":50,"rps":{"value":4544,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12052,"first_disqualified_conc":128,"lower_bound":false,"samples":27267},{"bound_ms":100,"rps":{"value":4544,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12052,"first_disqualified_conc":256,"lower_bound":false,"samples":27267},{"bound_ms":null,"rps":{"value":4544,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":12052,"first_disqualified_conc":null,"lower_bound":false,"samples":27267}],"sample_counts":{"gateway_c1_samples":9094,"direct_c1_samples":264951},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:bedrock:cohere","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":553,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":938,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":966,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":808.8659583856625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1236.29878304658,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9897773338096403,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":29461,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4909,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.016089066902006042,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":161.31640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1081.46484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1023.96875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1293.359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1176.1015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":70.10000993732086,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.3348033064043597,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":1683,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":932,"first_disqualified_conc":2,"lower_bound":false,"samples":10101},{"bound_ms":5,"rps":{"value":4732,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2267,"first_disqualified_conc":8,"lower_bound":false,"samples":28401},{"bound_ms":10,"rps":{"value":4965,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5080,"first_disqualified_conc":16,"lower_bound":false,"samples":29795},{"bound_ms":50,"rps":{"value":5048,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":10383,"first_disqualified_conc":128,"lower_bound":false,"samples":30292},{"bound_ms":100,"rps":{"value":5048,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":10383,"first_disqualified_conc":256,"lower_bound":false,"samples":30292},{"bound_ms":null,"rps":{"value":5048,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":10383,"first_disqualified_conc":null,"lower_bound":false,"samples":30292}],"sample_counts":{"gateway_c1_samples":10081,"direct_c1_samples":263966},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:openai:bedrock","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"bedrock","upstream_transport":"tls","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":708,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1098,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1126,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":955.841027699719,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1046.1990760184797,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9890819979689783,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":24910,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4151,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":17,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":159.95703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":981.90234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1138.0234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1138.05859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1131.03125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":92.38327794178917,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.7463544167999727,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":3880,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":3258,"first_disqualified_conc":8,"lower_bound":false,"samples":23291},{"bound_ms":10,"rps":{"value":4031,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":6052,"first_disqualified_conc":16,"lower_bound":false,"samples":24194},{"bound_ms":50,"rps":{"value":4078,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":27999,"first_disqualified_conc":128,"lower_bound":false,"samples":24475},{"bound_ms":100,"rps":{"value":4078,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":27999,"first_disqualified_conc":256,"lower_bound":false,"samples":24475},{"bound_ms":null,"rps":{"value":4078,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":27999,"first_disqualified_conc":null,"lower_bound":false,"samples":24475}],"sample_counts":{"gateway_c1_samples":7893,"direct_c1_samples":260423},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:openai-responses:bedrock","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"bedrock","upstream_transport":"tls","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":728,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1097,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":995.4637096774194,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1004.5569620253165,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9842016677515363,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":23808,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":3967,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":161.05078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":922.30078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1032.97265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1145.89453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1102.09375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":104.23628620670526,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.2653526376532094,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":3740,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":3132,"first_disqualified_conc":8,"lower_bound":false,"samples":22445},{"bound_ms":10,"rps":{"value":3865,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":6083,"first_disqualified_conc":16,"lower_bound":false,"samples":23201},{"bound_ms":50,"rps":{"value":3918,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":28074,"first_disqualified_conc":128,"lower_bound":false,"samples":23511},{"bound_ms":100,"rps":{"value":3918,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":28074,"first_disqualified_conc":256,"lower_bound":false,"samples":23511},{"bound_ms":null,"rps":{"value":3918,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":28074,"first_disqualified_conc":null,"lower_bound":false,"samples":23511}],"sample_counts":{"gateway_c1_samples":7776,"direct_c1_samples":260884},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:anthropic:bedrock","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"bedrock","upstream_transport":"tls","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":726,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1099,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1129,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1001.5123508654008,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":998.489932885906,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9899525457344274,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":23804,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":3966,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":20,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.03297765081498908,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":160.77734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":950.16015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1053.96875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1191.13671875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1138.20703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":99.68931391092528,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.3528148459708091,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":3426,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":3835,"first_disqualified_conc":8,"lower_bound":false,"samples":20565},{"bound_ms":10,"rps":{"value":3624,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":6912,"first_disqualified_conc":16,"lower_bound":false,"samples":21752},{"bound_ms":50,"rps":{"value":3706,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":30259,"first_disqualified_conc":64,"lower_bound":false,"samples":22241},{"bound_ms":100,"rps":{"value":3706,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":30259,"first_disqualified_conc":256,"lower_bound":false,"samples":22241},{"bound_ms":null,"rps":{"value":3706,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":30259,"first_disqualified_conc":null,"lower_bound":false,"samples":22241}],"sample_counts":{"gateway_c1_samples":7777,"direct_c1_samples":244991},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:gemini:bedrock","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"bedrock","upstream_transport":"tls","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":732,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1096,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":29,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1022.8790555387995,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":977.6326874473464,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9859755787262404,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":23209,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":3867,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.01930285665043733,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":159.62890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":951.703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1257.5859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1257.6171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1150.5234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":113.76387607449239,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.38908520290504295,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":3839,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":3321,"first_disqualified_conc":8,"lower_bound":false,"samples":23038},{"bound_ms":10,"rps":{"value":3970,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":6060,"first_disqualified_conc":16,"lower_bound":false,"samples":23829},{"bound_ms":50,"rps":{"value":4018,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":29568,"first_disqualified_conc":128,"lower_bound":false,"samples":24117},{"bound_ms":100,"rps":{"value":4018,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":29568,"first_disqualified_conc":256,"lower_bound":false,"samples":24117},{"bound_ms":null,"rps":{"value":4018,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":29568,"first_disqualified_conc":null,"lower_bound":false,"samples":24117}],"sample_counts":{"gateway_c1_samples":7747,"direct_c1_samples":257445},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:cohere:bedrock","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"bedrock","upstream_transport":"tls","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":692,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1061,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1090,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":29,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":938.6052056427578,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1065.41066892464,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9810888224468881,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":25165,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4193,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":162.12890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1001.58203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1040.97265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1135.625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1107.0546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":66.49416518170766,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.7986957676605473,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4119,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2685,"first_disqualified_conc":8,"lower_bound":false,"samples":24719},{"bound_ms":10,"rps":{"value":4263,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5666,"first_disqualified_conc":16,"lower_bound":false,"samples":25581},{"bound_ms":50,"rps":{"value":4275,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":25291,"first_disqualified_conc":128,"lower_bound":false,"samples":25658},{"bound_ms":100,"rps":{"value":4275,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":25291,"first_disqualified_conc":256,"lower_bound":false,"samples":25658},{"bound_ms":null,"rps":{"value":4275,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":25291,"first_disqualified_conc":null,"lower_bound":false,"samples":25658}],"sample_counts":{"gateway_c1_samples":8190,"direct_c1_samples":262997},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7:bedrock:bedrock","gateway_id":"bifrost","gateway_build":"maximhq/bifrost:v2.5.0@sha256:445bc49544bc64971b63581a93c6ca5716a03ccf27a58ced4c285cbafc133e78","run_id":"run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","measured_at":"2026-11-11T03:17:59Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"bedrock","upstream_transport":"tls","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"outcome":"pass","boxes":34,"per_box":"cell_runs[].rig.box_qualify"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":694,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1052,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1079,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":956.7628385437596,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1045.1910961780766,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.988885545960124,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":24886,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4147,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.03359318492325002,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":160.0234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":964.095703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1242.09765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1242.140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1119.36328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":61.39766170077196,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.7829862318956637,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":3832,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":3231,"first_disqualified_conc":8,"lower_bound":false,"samples":22999},{"bound_ms":10,"rps":{"value":3972,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":6449,"first_disqualified_conc":16,"lower_bound":false,"samples":23835},{"bound_ms":50,"rps":{"value":4035,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":27153,"first_disqualified_conc":128,"lower_bound":false,"samples":24214},{"bound_ms":100,"rps":{"value":4035,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":27153,"first_disqualified_conc":256,"lower_bound":false,"samples":24214},{"bound_ms":null,"rps":{"value":4035,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":27153,"first_disqualified_conc":null,"lower_bound":false,"samples":24214}],"sample_counts":{"gateway_c1_samples":8133,"direct_c1_samples":263582},"provenance":{"source_url":"https://onthebench.ai/gateways/bifrost","raw_result_url":"https://onthebench.ai/api/v1/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-bifrost-c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c253d4668350f1793f587661b425702d5448ca78d04ec9144589514b6064ebd7"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:cohere:anthropic","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":99,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":111,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":139,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":88.9270714012982,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":11245.16960068699,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9670695986523425,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":261900,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":43646,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.42578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":24.2890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":16.3984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":24.6875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":24.3125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.523499390824377,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.00400786510409457,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":46645,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":521,"first_disqualified_conc":32,"lower_bound":false,"samples":279903},{"bound_ms":5,"rps":{"value":47855,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4312,"first_disqualified_conc":256,"lower_bound":false,"samples":287174},{"bound_ms":10,"rps":{"value":47855,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4312,"first_disqualified_conc":256,"lower_bound":false,"samples":287174},{"bound_ms":50,"rps":{"value":47855,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4312,"first_disqualified_conc":1024,"lower_bound":false,"samples":287174},{"bound_ms":100,"rps":{"value":47855,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4312,"first_disqualified_conc":2048,"lower_bound":false,"samples":287174},{"bound_ms":null,"rps":{"value":47855,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4312,"first_disqualified_conc":16384,"lower_bound":false,"samples":287174}],"sample_counts":{"gateway_c1_samples":49012,"direct_c1_samples":264547},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:openai:anthropic","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":101,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":115,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":143,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":89.25547310404286,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":11203.794739111685,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9628878805359811,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":259816,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":43297,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":154,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":248,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21152us) came in under the mock's (21179us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":41099.647972885796,"certified":true,"suppressed":false,"headroom":0.5267899215274668,"rig_ceiling":78019.04761904762,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1536,"certified":true,"suppressed":false,"headroom":0.5267899215274668,"rig_ceiling":78019.04761904762,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.42578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":20.10546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.3671875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":20.6796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":20.47265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.14433783129212033,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.0040077451530521075,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":46006,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":530,"first_disqualified_conc":32,"lower_bound":false,"samples":276061},{"bound_ms":5,"rps":{"value":47478,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4355,"first_disqualified_conc":256,"lower_bound":false,"samples":284911},{"bound_ms":10,"rps":{"value":47478,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4355,"first_disqualified_conc":256,"lower_bound":false,"samples":284911},{"bound_ms":50,"rps":{"value":47478,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4355,"first_disqualified_conc":1024,"lower_bound":false,"samples":284911},{"bound_ms":100,"rps":{"value":47478,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4355,"first_disqualified_conc":2048,"lower_bound":false,"samples":284911},{"bound_ms":null,"rps":{"value":47478,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4355,"first_disqualified_conc":16384,"lower_bound":false,"samples":284911}],"sample_counts":{"gateway_c1_samples":47933,"direct_c1_samples":264624,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1792,"http_status_counts":{"200":1792},"stream_errors_no_response":0},{"conc":1664,"http_status_counts":{"200":1664},"stream_errors_no_response":0},{"conc":1600,"http_status_counts":{"200":1600},"stream_errors_no_response":0},{"conc":1568,"http_status_counts":{"200":1568},"stream_errors_no_response":0},{"conc":1552,"http_status_counts":{"200":1552},"stream_errors_no_response":0},{"conc":1544,"http_status_counts":{"200":1544},"stream_errors_no_response":0},{"conc":1540,"http_status_counts":{"200":1540},"stream_errors_no_response":0},{"conc":1538,"http_status_counts":{"200":1538},"stream_errors_no_response":0},{"conc":1537,"http_status_counts":{"200":1537},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:openai-responses:anthropic","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":150,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":166,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":193,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":136.43061454450898,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":7329.733163913596,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9804974961569917,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":173055,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":28839,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.4453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":20.8671875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.51953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":21.234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":20.78515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.13133944271705947,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004007090231122238,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":29647,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":828,"first_disqualified_conc":32,"lower_bound":false,"samples":177901},{"bound_ms":5,"rps":{"value":29773,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1666,"first_disqualified_conc":128,"lower_bound":false,"samples":178657},{"bound_ms":10,"rps":{"value":29773,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1666,"first_disqualified_conc":256,"lower_bound":false,"samples":178657},{"bound_ms":50,"rps":{"value":29773,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1666,"first_disqualified_conc":1024,"lower_bound":false,"samples":178657},{"bound_ms":100,"rps":{"value":29773,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1666,"first_disqualified_conc":2048,"lower_bound":false,"samples":178657},{"bound_ms":null,"rps":{"value":29773,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1666,"first_disqualified_conc":16384,"lower_bound":false,"samples":178657}],"sample_counts":{"gateway_c1_samples":34460,"direct_c1_samples":264688},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:anthropic:anthropic","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":85,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":96,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":124,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":74.1203039447186,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":13491.579861111111,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9565418741058359,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":310846,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":51801,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":229,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":2,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21139us) came in under the mock's (21146us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":43414.849772828435,"certified":true,"suppressed":false,"headroom":0.5033744728519197,"rig_ceiling":86247.61904761905,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1698,"certified":true,"suppressed":false,"headroom":0.5033744728519197,"rig_ceiling":86247.61904761905,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.44140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":22.20703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":16.39453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":23.2734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":22.98046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.58438609430649,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004008955053417264,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":57786,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":855,"first_disqualified_conc":64,"lower_bound":false,"samples":346749},{"bound_ms":5,"rps":{"value":58879,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":3520,"first_disqualified_conc":256,"lower_bound":false,"samples":353323},{"bound_ms":10,"rps":{"value":58879,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":3520,"first_disqualified_conc":512,"lower_bound":false,"samples":353323},{"bound_ms":50,"rps":{"value":58879,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":3520,"first_disqualified_conc":1024,"lower_bound":false,"samples":353323},{"bound_ms":100,"rps":{"value":58879,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":3520,"first_disqualified_conc":2048,"lower_bound":false,"samples":353323},{"bound_ms":null,"rps":{"value":58879,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":3520,"first_disqualified_conc":16384,"lower_bound":false,"samples":353323}],"sample_counts":{"gateway_c1_samples":55235,"direct_c1_samples":261103,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1792,"http_status_counts":{"200":1792},"stream_errors_no_response":0},{"conc":1664,"http_status_counts":{"200":1664},"stream_errors_no_response":0},{"conc":1728,"http_status_counts":{"200":1728},"stream_errors_no_response":0},{"conc":1696,"http_status_counts":{"200":1696},"stream_errors_no_response":0},{"conc":1712,"http_status_counts":{"200":1712},"stream_errors_no_response":0},{"conc":1704,"http_status_counts":{"200":1704},"stream_errors_no_response":0},{"conc":1700,"http_status_counts":{"200":1700},"stream_errors_no_response":0},{"conc":1698,"http_status_counts":{"200":1698},"stream_errors_no_response":0},{"conc":1699,"http_status_counts":{"200":1699},"stream_errors_no_response":0},{"conc":1698,"http_status_counts":{"200":1698},"stream_errors_no_response":0},{"conc":1698,"http_status_counts":{"200":1698},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:gemini:anthropic","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":105,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":119,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":146,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":93.11083747781171,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":10739.888364104767,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9671513679614686,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":250132,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":41684,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.42578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":22.046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":16.3203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":22.44140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":22.28125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.15884976848051902,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004007644983783626,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":44397,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":552,"first_disqualified_conc":32,"lower_bound":false,"samples":266417},{"bound_ms":5,"rps":{"value":45198,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1097,"first_disqualified_conc":256,"lower_bound":false,"samples":271212},{"bound_ms":10,"rps":{"value":45198,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1097,"first_disqualified_conc":256,"lower_bound":false,"samples":271212},{"bound_ms":50,"rps":{"value":45198,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1097,"first_disqualified_conc":1024,"lower_bound":false,"samples":271212},{"bound_ms":100,"rps":{"value":45198,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1097,"first_disqualified_conc":2048,"lower_bound":false,"samples":271212},{"bound_ms":null,"rps":{"value":45198,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1097,"first_disqualified_conc":16384,"lower_bound":false,"samples":271212}],"sample_counts":{"gateway_c1_samples":46775,"direct_c1_samples":263231},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:bedrock:anthropic","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":106,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":120,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":147,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":93.7171234284747,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":10670.408602150537,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9654400330958657,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":248087,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":41343,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.44140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":26.2734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":16.046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":26.7578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":26.52734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.44241288485165836,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.00400619555458917,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":43964,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":557,"first_disqualified_conc":32,"lower_bound":false,"samples":263811},{"bound_ms":5,"rps":{"value":45086,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2234,"first_disqualified_conc":256,"lower_bound":false,"samples":270545},{"bound_ms":10,"rps":{"value":45086,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2234,"first_disqualified_conc":256,"lower_bound":false,"samples":270545},{"bound_ms":50,"rps":{"value":45086,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2234,"first_disqualified_conc":1024,"lower_bound":false,"samples":270545},{"bound_ms":100,"rps":{"value":45086,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2234,"first_disqualified_conc":2048,"lower_bound":false,"samples":270545},{"bound_ms":null,"rps":{"value":45086,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2234,"first_disqualified_conc":16384,"lower_bound":false,"samples":270545}],"sample_counts":{"gateway_c1_samples":46122,"direct_c1_samples":266395},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:openai-responses:bedrock","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":171,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":192,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":219,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":157.14485728953457,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":6363.555366991939,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9790907506131737,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":149989,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":24995,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.44140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":23.21875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":14.2734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":23.453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":23.2265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.22186986479277265,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004010617791379746,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":25676,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":953,"first_disqualified_conc":32,"lower_bound":false,"samples":154074},{"bound_ms":5,"rps":{"value":25676,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":953,"first_disqualified_conc":128,"lower_bound":false,"samples":154074},{"bound_ms":10,"rps":{"value":25676,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":953,"first_disqualified_conc":256,"lower_bound":false,"samples":154074},{"bound_ms":50,"rps":{"value":25676,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":953,"first_disqualified_conc":1024,"lower_bound":false,"samples":154074},{"bound_ms":100,"rps":{"value":25676,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":953,"first_disqualified_conc":2048,"lower_bound":false,"samples":154074},{"bound_ms":null,"rps":{"value":25676,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":953,"first_disqualified_conc":16384,"lower_bound":false,"samples":154074}],"sample_counts":{"gateway_c1_samples":30814,"direct_c1_samples":264867},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:gemini:bedrock","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":129,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":148,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":175,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":115.67683836466455,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":8644.772922022279,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9693903070517332,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":201769,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":33624,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.44921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":21.8125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.04296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":22.1484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":21.80859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.34623519901631217,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.003998549651253959,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":34935,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":701,"first_disqualified_conc":32,"lower_bound":false,"samples":209641},{"bound_ms":5,"rps":{"value":35197,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2844,"first_disqualified_conc":128,"lower_bound":false,"samples":211190},{"bound_ms":10,"rps":{"value":35197,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2844,"first_disqualified_conc":256,"lower_bound":false,"samples":211190},{"bound_ms":50,"rps":{"value":35197,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2844,"first_disqualified_conc":1024,"lower_bound":false,"samples":211190},{"bound_ms":100,"rps":{"value":35197,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2844,"first_disqualified_conc":2048,"lower_bound":false,"samples":211190},{"bound_ms":null,"rps":{"value":35197,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2844,"first_disqualified_conc":16384,"lower_bound":false,"samples":211190}],"sample_counts":{"gateway_c1_samples":39387,"direct_c1_samples":270577},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:cohere:bedrock","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":120,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":136,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":163,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":109.89425447756621,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":9099.656799656801,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.968107606611598,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":212113,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":35348,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":20.296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.1796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":20.45703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":20.18359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.34446157219296425,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004008712905503679,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":36699,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":668,"first_disqualified_conc":32,"lower_bound":false,"samples":220217},{"bound_ms":5,"rps":{"value":36731,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1356,"first_disqualified_conc":128,"lower_bound":false,"samples":220409},{"bound_ms":10,"rps":{"value":36731,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1356,"first_disqualified_conc":256,"lower_bound":false,"samples":220409},{"bound_ms":50,"rps":{"value":36731,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1356,"first_disqualified_conc":1024,"lower_bound":false,"samples":220409},{"bound_ms":100,"rps":{"value":36731,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1356,"first_disqualified_conc":2048,"lower_bound":false,"samples":220409},{"bound_ms":null,"rps":{"value":36731,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1356,"first_disqualified_conc":16384,"lower_bound":false,"samples":220409}],"sample_counts":{"gateway_c1_samples":41840,"direct_c1_samples":268504},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:bedrock:bedrock","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":110,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":127,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":154,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":101.07769859203894,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":9893.379191745486,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9659932319078792,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":230120,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":38348,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":20.45703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.17578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":20.74609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":20.6484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.062346202266867504,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004010021897584423,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":39956,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":612,"first_disqualified_conc":32,"lower_bound":false,"samples":239762},{"bound_ms":5,"rps":{"value":40107,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1241,"first_disqualified_conc":128,"lower_bound":false,"samples":240666},{"bound_ms":10,"rps":{"value":40107,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1241,"first_disqualified_conc":256,"lower_bound":false,"samples":240666},{"bound_ms":50,"rps":{"value":40107,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1241,"first_disqualified_conc":1024,"lower_bound":false,"samples":240666},{"bound_ms":100,"rps":{"value":40107,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1241,"first_disqualified_conc":2048,"lower_bound":false,"samples":240666},{"bound_ms":null,"rps":{"value":40107,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1241,"first_disqualified_conc":16384,"lower_bound":false,"samples":240666}],"sample_counts":{"gateway_c1_samples":44888,"direct_c1_samples":266888},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:openai:bedrock","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":142,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":170,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":111.37322181714204,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":8978.818998716302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9706288233077657,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":209835,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":34969,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.42578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":19.92578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.17578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":20.3671875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":19.97265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.07346962001177246,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004007962647591097,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":36468,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":672,"first_disqualified_conc":32,"lower_bound":false,"samples":218834},{"bound_ms":5,"rps":{"value":36523,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1368,"first_disqualified_conc":128,"lower_bound":false,"samples":219157},{"bound_ms":10,"rps":{"value":36523,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1368,"first_disqualified_conc":256,"lower_bound":false,"samples":219157},{"bound_ms":50,"rps":{"value":36523,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1368,"first_disqualified_conc":1024,"lower_bound":false,"samples":219157},{"bound_ms":100,"rps":{"value":36523,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1368,"first_disqualified_conc":2048,"lower_bound":false,"samples":219157},{"bound_ms":null,"rps":{"value":36523,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1368,"first_disqualified_conc":16384,"lower_bound":false,"samples":219157}],"sample_counts":{"gateway_c1_samples":40371,"direct_c1_samples":265424},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:anthropic:bedrock","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":123,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":140,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":168,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":111.5167036159315,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":8967.26649528706,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9693346432154761,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":209296,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":34878,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":20.265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.97265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":21.0703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":21.0078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.559052607494058,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.003997921959096089,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":36181,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":678,"first_disqualified_conc":32,"lower_bound":false,"samples":217103},{"bound_ms":5,"rps":{"value":36453,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2759,"first_disqualified_conc":128,"lower_bound":false,"samples":218731},{"bound_ms":10,"rps":{"value":36453,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2759,"first_disqualified_conc":256,"lower_bound":false,"samples":218731},{"bound_ms":50,"rps":{"value":36453,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2759,"first_disqualified_conc":1024,"lower_bound":false,"samples":218731},{"bound_ms":100,"rps":{"value":36453,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2759,"first_disqualified_conc":2048,"lower_bound":false,"samples":218731},{"bound_ms":null,"rps":{"value":36453,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2759,"first_disqualified_conc":16384,"lower_bound":false,"samples":218731}],"sample_counts":{"gateway_c1_samples":40751,"direct_c1_samples":263427},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:openai:openai","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":78,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":89,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":117,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":68.74185107722418,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":14547.178819444445,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9565490360420432,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":335167,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":55854,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":118,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":254,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21068us) came in under the mock's (21071us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21128us) came in under the mock's (21237us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":39240.21276535891,"certified":true,"suppressed":false,"headroom":0.5320535046955948,"rig_ceiling":73752.38095238095,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1452,"certified":true,"suppressed":false,"headroom":0.5320535046955948,"rig_ceiling":73752.38095238095,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.42578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":21.4921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":22.3515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":22.26953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.39135543613149704,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004006197677784261,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":62231,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":795,"first_disqualified_conc":64,"lower_bound":false,"samples":373417},{"bound_ms":5,"rps":{"value":62587,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":3317,"first_disqualified_conc":256,"lower_bound":false,"samples":375578},{"bound_ms":10,"rps":{"value":62587,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":3317,"first_disqualified_conc":512,"lower_bound":false,"samples":375578},{"bound_ms":50,"rps":{"value":62587,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":3317,"first_disqualified_conc":1024,"lower_bound":false,"samples":375578},{"bound_ms":100,"rps":{"value":62587,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":3317,"first_disqualified_conc":2048,"lower_bound":false,"samples":375578},{"bound_ms":null,"rps":{"value":62587,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":3317,"first_disqualified_conc":16384,"lower_bound":false,"samples":375578}],"sample_counts":{"gateway_c1_samples":59266,"direct_c1_samples":266330,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1280,"http_status_counts":{"200":1280},"stream_errors_no_response":0},{"conc":1408,"http_status_counts":{"200":1408},"stream_errors_no_response":0},{"conc":1472,"http_status_counts":{"200":1472},"stream_errors_no_response":0},{"conc":1440,"http_status_counts":{"200":1440},"stream_errors_no_response":0},{"conc":1456,"http_status_counts":{"200":1456},"stream_errors_no_response":0},{"conc":1448,"http_status_counts":{"200":1448},"stream_errors_no_response":0},{"conc":1452,"http_status_counts":{"200":1452},"stream_errors_no_response":0},{"conc":1454,"http_status_counts":{"200":1454},"stream_errors_no_response":0},{"conc":1453,"http_status_counts":{"200":1453},"stream_errors_no_response":0},{"conc":1452,"http_status_counts":{"200":1452},"stream_errors_no_response":0},{"conc":1452,"http_status_counts":{"200":1452},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:cohere:openai","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":98,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":111,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":138,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":86.89654138269717,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":11507.937877480586,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9625241601910512,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":266754,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":44454,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.42578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":20.19140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.3046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":20.4140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":20.15234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":276,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.03774159125433938,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004005444131551088,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":47293,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":518,"first_disqualified_conc":32,"lower_bound":false,"samples":283788},{"bound_ms":5,"rps":{"value":48545,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4242,"first_disqualified_conc":256,"lower_bound":false,"samples":291307},{"bound_ms":10,"rps":{"value":48545,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4242,"first_disqualified_conc":256,"lower_bound":false,"samples":291307},{"bound_ms":50,"rps":{"value":48545,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4242,"first_disqualified_conc":1024,"lower_bound":false,"samples":291307},{"bound_ms":100,"rps":{"value":48545,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4242,"first_disqualified_conc":2048,"lower_bound":false,"samples":291307},{"bound_ms":null,"rps":{"value":48545,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4242,"first_disqualified_conc":16384,"lower_bound":false,"samples":291307}],"sample_counts":{"gateway_c1_samples":49364,"direct_c1_samples":265547},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:anthropic:openai","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":100,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":113,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":141,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":88.98443348240849,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":11237.920621225194,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9625705250300515,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":260495,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":43410,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":118,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":213,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21131us) came in under the mock's (21181us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":44699.8581954769,"certified":true,"suppressed":false,"headroom":0.5466015268468642,"rig_ceiling":81777.77777777778,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1610,"certified":true,"suppressed":false,"headroom":0.5466015268468642,"rig_ceiling":81777.77777777778,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":23.484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":16.34375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":23.8359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":23.72265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.3003414899674397,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004007447817674569,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":46675,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":525,"first_disqualified_conc":32,"lower_bound":false,"samples":280076},{"bound_ms":5,"rps":{"value":47965,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2096,"first_disqualified_conc":256,"lower_bound":false,"samples":287798},{"bound_ms":10,"rps":{"value":47965,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2096,"first_disqualified_conc":256,"lower_bound":false,"samples":287798},{"bound_ms":50,"rps":{"value":47965,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2096,"first_disqualified_conc":1024,"lower_bound":false,"samples":287798},{"bound_ms":100,"rps":{"value":47965,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2096,"first_disqualified_conc":2048,"lower_bound":false,"samples":287798},{"bound_ms":null,"rps":{"value":47965,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2096,"first_disqualified_conc":16384,"lower_bound":false,"samples":287798}],"sample_counts":{"gateway_c1_samples":48328,"direct_c1_samples":264163,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1792,"http_status_counts":{"200":1792},"stream_errors_no_response":0},{"conc":1664,"http_status_counts":{"200":1664},"stream_errors_no_response":0},{"conc":1600,"http_status_counts":{"200":1600},"stream_errors_no_response":0},{"conc":1632,"http_status_counts":{"200":1632},"stream_errors_no_response":0},{"conc":1616,"http_status_counts":{"200":1616},"stream_errors_no_response":0},{"conc":1608,"http_status_counts":{"200":1608},"stream_errors_no_response":0},{"conc":1612,"http_status_counts":{"200":1612},"stream_errors_no_response":0},{"conc":1610,"http_status_counts":{"200":1610},"stream_errors_no_response":0},{"conc":1611,"http_status_counts":{"200":1611},"stream_errors_no_response":0},{"conc":1610,"http_status_counts":{"200":1610},"stream_errors_no_response":0},{"conc":1610,"http_status_counts":{"200":1610},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:gemini:openai","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":104,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":118,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":145,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":92.7242504956642,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":10784.665226781859,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9613428390516816,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":249665,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":41606,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.42578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":21.84375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":16.44921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":22.5078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":22.48046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.5719859049680833,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004007133333991704,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":45104,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":544,"first_disqualified_conc":32,"lower_bound":false,"samples":270658},{"bound_ms":5,"rps":{"value":46080,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1067,"first_disqualified_conc":256,"lower_bound":false,"samples":276504},{"bound_ms":10,"rps":{"value":46080,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1067,"first_disqualified_conc":256,"lower_bound":false,"samples":276504},{"bound_ms":50,"rps":{"value":46080,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1067,"first_disqualified_conc":1024,"lower_bound":false,"samples":276504},{"bound_ms":100,"rps":{"value":46080,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1067,"first_disqualified_conc":2048,"lower_bound":false,"samples":276504},{"bound_ms":null,"rps":{"value":46080,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1067,"first_disqualified_conc":16384,"lower_bound":false,"samples":276504}],"sample_counts":{"gateway_c1_samples":46858,"direct_c1_samples":265285},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:openai-responses:openai","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":144,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":159,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":186,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":134.46200964346505,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":7437.044877222692,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9809619486700766,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":175663,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":29273,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.43359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":21.12890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.37890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":21.75390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":21.44921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.3702076945603973,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.0040077564965903,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":30322,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":811,"first_disqualified_conc":32,"lower_bound":false,"samples":181950},{"bound_ms":5,"rps":{"value":30374,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1631,"first_disqualified_conc":128,"lower_bound":false,"samples":182256},{"bound_ms":10,"rps":{"value":30374,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1631,"first_disqualified_conc":256,"lower_bound":false,"samples":182256},{"bound_ms":50,"rps":{"value":30374,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1631,"first_disqualified_conc":1024,"lower_bound":false,"samples":182256},{"bound_ms":100,"rps":{"value":30374,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1631,"first_disqualified_conc":2048,"lower_bound":false,"samples":182256},{"bound_ms":null,"rps":{"value":30374,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1631,"first_disqualified_conc":16384,"lower_bound":false,"samples":182256}],"sample_counts":{"gateway_c1_samples":35866,"direct_c1_samples":266530},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:bedrock:openai","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":103,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":115,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":143,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":92.1217041122492,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":10855.205183585314,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9612699976255128,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":251298,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":41879,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.41796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":20.6875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.16796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":21.0546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":20.796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":192,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.050457226911487886,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.00400587441821304,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":45153,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":543,"first_disqualified_conc":32,"lower_bound":false,"samples":270945},{"bound_ms":5,"rps":{"value":46330,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2161,"first_disqualified_conc":256,"lower_bound":false,"samples":277993},{"bound_ms":10,"rps":{"value":46330,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2161,"first_disqualified_conc":256,"lower_bound":false,"samples":277993},{"bound_ms":50,"rps":{"value":46330,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2161,"first_disqualified_conc":1024,"lower_bound":false,"samples":277993},{"bound_ms":100,"rps":{"value":46330,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2161,"first_disqualified_conc":2048,"lower_bound":false,"samples":277993},{"bound_ms":null,"rps":{"value":46330,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2161,"first_disqualified_conc":16384,"lower_bound":false,"samples":277993}],"sample_counts":{"gateway_c1_samples":47501,"direct_c1_samples":263962},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:anthropic:cohere","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":99,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":112,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":139,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":87.3895351692818,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":11443.017725897103,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.960485712307784,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":264677,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":44107,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.4140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":23.14453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.30078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":24.06640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":23.8515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.45313705745113203,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004009000486707295,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":47488,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":516,"first_disqualified_conc":32,"lower_bound":false,"samples":284957},{"bound_ms":5,"rps":{"value":48799,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2045,"first_disqualified_conc":256,"lower_bound":false,"samples":292804},{"bound_ms":10,"rps":{"value":48799,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2045,"first_disqualified_conc":256,"lower_bound":false,"samples":292804},{"bound_ms":50,"rps":{"value":48799,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2045,"first_disqualified_conc":1024,"lower_bound":false,"samples":292804},{"bound_ms":100,"rps":{"value":48799,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2045,"first_disqualified_conc":2048,"lower_bound":false,"samples":292804},{"bound_ms":null,"rps":{"value":48799,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2045,"first_disqualified_conc":16384,"lower_bound":false,"samples":292804}],"sample_counts":{"gateway_c1_samples":48830,"direct_c1_samples":265034},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:openai:cohere","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":96,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":109,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":136,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":87.43816361003861,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":11436.653730056058,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9628483639273625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":265216,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":44197,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.4375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":21.41015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":16.34765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":21.8046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":21.80078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.05080811740817498,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004008281792599479,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":47491,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":516,"first_disqualified_conc":32,"lower_bound":false,"samples":284980},{"bound_ms":5,"rps":{"value":49018,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4223,"first_disqualified_conc":256,"lower_bound":false,"samples":294145},{"bound_ms":10,"rps":{"value":49018,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4223,"first_disqualified_conc":256,"lower_bound":false,"samples":294145},{"bound_ms":50,"rps":{"value":49018,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4223,"first_disqualified_conc":1024,"lower_bound":false,"samples":294145},{"bound_ms":100,"rps":{"value":49018,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4223,"first_disqualified_conc":2048,"lower_bound":false,"samples":294145},{"bound_ms":null,"rps":{"value":49018,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4223,"first_disqualified_conc":16384,"lower_bound":false,"samples":294145}],"sample_counts":{"gateway_c1_samples":49997,"direct_c1_samples":264312},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:openai-responses:cohere","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":142,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":157,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":184,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":133.22123414435967,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":7506.3108852181285,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9804956805820538,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":177224,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":29533,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":23.77734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":24.53125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":24,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.6699562011229396,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.00400867981597507,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":30560,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":804,"first_disqualified_conc":32,"lower_bound":false,"samples":183378},{"bound_ms":5,"rps":{"value":30675,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1617,"first_disqualified_conc":128,"lower_bound":false,"samples":184074},{"bound_ms":10,"rps":{"value":30675,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1617,"first_disqualified_conc":256,"lower_bound":false,"samples":184074},{"bound_ms":50,"rps":{"value":30675,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1617,"first_disqualified_conc":1024,"lower_bound":false,"samples":184074},{"bound_ms":100,"rps":{"value":30675,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1617,"first_disqualified_conc":2048,"lower_bound":false,"samples":184074},{"bound_ms":null,"rps":{"value":30675,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1617,"first_disqualified_conc":16384,"lower_bound":false,"samples":184074}],"sample_counts":{"gateway_c1_samples":36123,"direct_c1_samples":267803},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:gemini:cohere","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":99,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":112,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":139,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":91.58016330147478,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":10919.39524838013,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9613093332432712,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":252784,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":42125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.44921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":20.9296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.7890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":21.64453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":21.359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.26857538616502413,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.00400832060437658,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":45456,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":541,"first_disqualified_conc":32,"lower_bound":false,"samples":272767},{"bound_ms":5,"rps":{"value":46559,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4430,"first_disqualified_conc":256,"lower_bound":false,"samples":279399},{"bound_ms":10,"rps":{"value":46559,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4430,"first_disqualified_conc":256,"lower_bound":false,"samples":279399},{"bound_ms":50,"rps":{"value":46559,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4430,"first_disqualified_conc":1024,"lower_bound":false,"samples":279399},{"bound_ms":100,"rps":{"value":46559,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4430,"first_disqualified_conc":2048,"lower_bound":false,"samples":279399},{"bound_ms":null,"rps":{"value":46559,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":4430,"first_disqualified_conc":16384,"lower_bound":false,"samples":279399}],"sample_counts":{"gateway_c1_samples":48806,"direct_c1_samples":265602},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:cohere:cohere","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":78,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":89,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":116,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":69.33806822456626,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":14422.092013888889,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9565812625872341,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":332285,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":55375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.44140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":21.1015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.60546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":21.70703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":21.43359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.03836450603782764,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.0040093104292098095,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":61466,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":806,"first_disqualified_conc":64,"lower_bound":false,"samples":368809},{"bound_ms":5,"rps":{"value":62630,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":1596,"first_disqualified_conc":256,"lower_bound":false,"samples":375843},{"bound_ms":10,"rps":{"value":62630,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":1596,"first_disqualified_conc":512,"lower_bound":false,"samples":375843},{"bound_ms":50,"rps":{"value":62630,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":1596,"first_disqualified_conc":1024,"lower_bound":false,"samples":375843},{"bound_ms":100,"rps":{"value":62630,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":1596,"first_disqualified_conc":2048,"lower_bound":false,"samples":375843},{"bound_ms":null,"rps":{"value":62630,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":1596,"first_disqualified_conc":16384,"lower_bound":false,"samples":375843}],"sample_counts":{"gateway_c1_samples":59104,"direct_c1_samples":267546},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:bedrock:cohere","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":103,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":115,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":142,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":91.37204355492997,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":10944.266551276503,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.959609722336054,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":252922,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":42148,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.42578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":22.61328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.52734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":23.04296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":22.73828125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.055680741624084824,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004009476950966018,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":45878,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":535,"first_disqualified_conc":32,"lower_bound":false,"samples":275299},{"bound_ms":5,"rps":{"value":46932,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2134,"first_disqualified_conc":256,"lower_bound":false,"samples":281601},{"bound_ms":10,"rps":{"value":46932,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2134,"first_disqualified_conc":256,"lower_bound":false,"samples":281601},{"bound_ms":50,"rps":{"value":46932,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2134,"first_disqualified_conc":1024,"lower_bound":false,"samples":281601},{"bound_ms":100,"rps":{"value":46932,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2134,"first_disqualified_conc":2048,"lower_bound":false,"samples":281601},{"bound_ms":null,"rps":{"value":46932,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2134,"first_disqualified_conc":16384,"lower_bound":false,"samples":281601}],"sample_counts":{"gateway_c1_samples":47542,"direct_c1_samples":266764},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:gemini:openai-responses","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":108,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":121,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":149,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":96.56705976414152,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":10355.498059508409,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9629616642250729,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":240144,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":40019,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.4296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":22.6484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.47265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":23.53125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":23.53125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":1.3511256752055836,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004005703794387902,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":42669,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":576,"first_disqualified_conc":32,"lower_bound":false,"samples":256046},{"bound_ms":5,"rps":{"value":43426,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2314,"first_disqualified_conc":256,"lower_bound":false,"samples":260577},{"bound_ms":10,"rps":{"value":43426,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2314,"first_disqualified_conc":256,"lower_bound":false,"samples":260577},{"bound_ms":50,"rps":{"value":43426,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2314,"first_disqualified_conc":1024,"lower_bound":false,"samples":260577},{"bound_ms":100,"rps":{"value":43426,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2314,"first_disqualified_conc":2048,"lower_bound":false,"samples":260577},{"bound_ms":null,"rps":{"value":43426,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2314,"first_disqualified_conc":16384,"lower_bound":false,"samples":260577}],"sample_counts":{"gateway_c1_samples":45574,"direct_c1_samples":262793},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:anthropic:openai-responses","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":105,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":118,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":146,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":93.28839579436239,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":10719.446845289542,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9609150390781307,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":248048,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":41335,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.41796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":21.109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.6328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":21.59375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":21.47265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":156,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.10793647291847215,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004006902860821863,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":44406,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":552,"first_disqualified_conc":32,"lower_bound":false,"samples":266467},{"bound_ms":5,"rps":{"value":45442,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2211,"first_disqualified_conc":256,"lower_bound":false,"samples":272657},{"bound_ms":10,"rps":{"value":45442,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2211,"first_disqualified_conc":256,"lower_bound":false,"samples":272657},{"bound_ms":50,"rps":{"value":45442,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2211,"first_disqualified_conc":1024,"lower_bound":false,"samples":272657},{"bound_ms":100,"rps":{"value":45442,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2211,"first_disqualified_conc":2048,"lower_bound":false,"samples":272657},{"bound_ms":null,"rps":{"value":45442,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2211,"first_disqualified_conc":16384,"lower_bound":false,"samples":272657}],"sample_counts":{"gateway_c1_samples":46499,"direct_c1_samples":259447},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:openai:openai-responses","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":104,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":117,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":145,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":92.93284658048088,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":10760.458081244598,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9608785321621404,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":248997,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":41494,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.41015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":21.265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.7734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":21.65234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":21.42578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":246,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.0694395082616266,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.0037262713635202575,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":44763,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":548,"first_disqualified_conc":32,"lower_bound":false,"samples":268603},{"bound_ms":5,"rps":{"value":45770,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2199,"first_disqualified_conc":256,"lower_bound":false,"samples":274629},{"bound_ms":10,"rps":{"value":45770,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2199,"first_disqualified_conc":256,"lower_bound":false,"samples":274629},{"bound_ms":50,"rps":{"value":45770,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2199,"first_disqualified_conc":1024,"lower_bound":false,"samples":274629},{"bound_ms":100,"rps":{"value":45770,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2199,"first_disqualified_conc":2048,"lower_bound":false,"samples":274629},{"bound_ms":null,"rps":{"value":45770,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2199,"first_disqualified_conc":16384,"lower_bound":false,"samples":274629}],"sample_counts":{"gateway_c1_samples":46929,"direct_c1_samples":263227},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:cohere:openai-responses","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":102,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":116,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":143,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":91.4781315037707,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":10931.574394463667,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9600230504922741,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":252738,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":42119,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.44140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":21.578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":16.15234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":22.04296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":21.6171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still falling - releasing memory, not leaking it when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.4738938001696382,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004006452487671492,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":45427,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":539,"first_disqualified_conc":32,"lower_bound":false,"samples":272592},{"bound_ms":5,"rps":{"value":46419,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2158,"first_disqualified_conc":256,"lower_bound":false,"samples":278528},{"bound_ms":10,"rps":{"value":46419,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2158,"first_disqualified_conc":256,"lower_bound":false,"samples":278528},{"bound_ms":50,"rps":{"value":46419,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2158,"first_disqualified_conc":1024,"lower_bound":false,"samples":278528},{"bound_ms":100,"rps":{"value":46419,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2158,"first_disqualified_conc":2048,"lower_bound":false,"samples":278528},{"bound_ms":null,"rps":{"value":46419,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2158,"first_disqualified_conc":16384,"lower_bound":false,"samples":278528}],"sample_counts":{"gateway_c1_samples":47752,"direct_c1_samples":268640},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:openai-responses:openai-responses","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":79,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":90,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":118,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":70.63658809637002,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":14156.969170646982,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9561170433364696,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":326035,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":54332,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.43359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":20.76953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.61328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":21.48828125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":21.4296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.6822956920901504,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.0040077286694608786,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":59818,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":827,"first_disqualified_conc":64,"lower_bound":false,"samples":358944},{"bound_ms":5,"rps":{"value":60832,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":3391,"first_disqualified_conc":256,"lower_bound":false,"samples":365044},{"bound_ms":10,"rps":{"value":60832,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":3391,"first_disqualified_conc":512,"lower_bound":false,"samples":365044},{"bound_ms":50,"rps":{"value":60832,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":3391,"first_disqualified_conc":1024,"lower_bound":false,"samples":365044},{"bound_ms":100,"rps":{"value":60832,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":3391,"first_disqualified_conc":2048,"lower_bound":false,"samples":365044},{"bound_ms":null,"rps":{"value":60832,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":3391,"first_disqualified_conc":16384,"lower_bound":false,"samples":365044}],"sample_counts":{"gateway_c1_samples":58401,"direct_c1_samples":264340},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:bedrock:openai-responses","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":110,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":123,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":151,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":97.03562869602078,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":10305.493079584774,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9600148000930796,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":238263,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":39706,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.4296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":22.62109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.9921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":23.48828125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":23.328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.5903300861400782,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.00401049189688419,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":42923,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":572,"first_disqualified_conc":32,"lower_bound":false,"samples":257569},{"bound_ms":5,"rps":{"value":43797,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2276,"first_disqualified_conc":256,"lower_bound":false,"samples":262796},{"bound_ms":10,"rps":{"value":43797,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2276,"first_disqualified_conc":256,"lower_bound":false,"samples":262796},{"bound_ms":50,"rps":{"value":43797,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2276,"first_disqualified_conc":1024,"lower_bound":false,"samples":262796},{"bound_ms":100,"rps":{"value":43797,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2276,"first_disqualified_conc":2048,"lower_bound":false,"samples":262796},{"bound_ms":null,"rps":{"value":43797,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2276,"first_disqualified_conc":16384,"lower_bound":false,"samples":262796}],"sample_counts":{"gateway_c1_samples":44818,"direct_c1_samples":263285},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:anthropic:gemini","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":101,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":113,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":141,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":90.32393642689249,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":11071.262386902197,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9637455765730645,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":256964,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":42822,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.44140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":20.82421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.73046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":21.046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":20.8359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.3469890680174716,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004007174851919233,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":45587,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":536,"first_disqualified_conc":32,"lower_bound":false,"samples":273548},{"bound_ms":5,"rps":{"value":46654,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2146,"first_disqualified_conc":256,"lower_bound":false,"samples":279944},{"bound_ms":10,"rps":{"value":46654,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2146,"first_disqualified_conc":256,"lower_bound":false,"samples":279944},{"bound_ms":50,"rps":{"value":46654,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2146,"first_disqualified_conc":1024,"lower_bound":false,"samples":279944},{"bound_ms":100,"rps":{"value":46654,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2146,"first_disqualified_conc":2048,"lower_bound":false,"samples":279944},{"bound_ms":null,"rps":{"value":46654,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2146,"first_disqualified_conc":16384,"lower_bound":false,"samples":279944}],"sample_counts":{"gateway_c1_samples":48133,"direct_c1_samples":261686},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:bedrock:gemini","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":104,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":117,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":144,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":93.4326391891728,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":10702.897923875433,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9600302873676964,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":247451,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":41236,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.41796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":21.5078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.63671875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":22.76171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":22.25390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.6514809803331223,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004004194147117859,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":44045,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":558,"first_disqualified_conc":32,"lower_bound":false,"samples":264301},{"bound_ms":5,"rps":{"value":45269,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2208,"first_disqualified_conc":256,"lower_bound":false,"samples":271622},{"bound_ms":10,"rps":{"value":45269,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2208,"first_disqualified_conc":256,"lower_bound":false,"samples":271622},{"bound_ms":50,"rps":{"value":45269,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2208,"first_disqualified_conc":1024,"lower_bound":false,"samples":271622},{"bound_ms":100,"rps":{"value":45269,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2208,"first_disqualified_conc":2048,"lower_bound":false,"samples":271622},{"bound_ms":null,"rps":{"value":45269,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2208,"first_disqualified_conc":16384,"lower_bound":false,"samples":271622}],"sample_counts":{"gateway_c1_samples":47051,"direct_c1_samples":267236},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:gemini:gemini","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":87,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":97,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":77.7347222921175,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":12864.264134656883,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9620017764898603,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":298065,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":49672,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":21.41796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.9140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":21.93359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":21.64453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.001814087129386375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004006939579821272,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":54786,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":902,"first_disqualified_conc":64,"lower_bound":false,"samples":328745},{"bound_ms":5,"rps":{"value":55374,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":1803,"first_disqualified_conc":256,"lower_bound":false,"samples":332269},{"bound_ms":10,"rps":{"value":55374,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":1803,"first_disqualified_conc":512,"lower_bound":false,"samples":332269},{"bound_ms":50,"rps":{"value":55374,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":1803,"first_disqualified_conc":1024,"lower_bound":false,"samples":332269},{"bound_ms":100,"rps":{"value":55374,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":1803,"first_disqualified_conc":2048,"lower_bound":false,"samples":332269},{"bound_ms":null,"rps":{"value":55374,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":1803,"first_disqualified_conc":16384,"lower_bound":false,"samples":332269}],"sample_counts":{"gateway_c1_samples":54333,"direct_c1_samples":264586},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:openai:gemini","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":99,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":111,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":138,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":89.81653259383485,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":11133.807675722293,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9629225241232044,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":258193,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":43027,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.41796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":22.40234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":14.94921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":22.7265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":22.7265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.07816815050166112,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.0040067574750589086,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":46221,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":530,"first_disqualified_conc":32,"lower_bound":false,"samples":277357},{"bound_ms":5,"rps":{"value":47238,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2122,"first_disqualified_conc":256,"lower_bound":false,"samples":283442},{"bound_ms":10,"rps":{"value":47238,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2122,"first_disqualified_conc":256,"lower_bound":false,"samples":283442},{"bound_ms":50,"rps":{"value":47238,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2122,"first_disqualified_conc":1024,"lower_bound":false,"samples":283442},{"bound_ms":100,"rps":{"value":47238,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2122,"first_disqualified_conc":2048,"lower_bound":false,"samples":283442},{"bound_ms":null,"rps":{"value":47238,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2122,"first_disqualified_conc":16384,"lower_bound":false,"samples":283442}],"sample_counts":{"gateway_c1_samples":48998,"direct_c1_samples":264583},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:openai-responses:gemini","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":148,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":162,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":190,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":136.82118207474457,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":7308.809826344769,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9805341744257635,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":172561,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":28757,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.42578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":21.734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.08203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":22.40625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":21.9921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.4292936032053184,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004007859077210462,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":29952,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":820,"first_disqualified_conc":32,"lower_bound":false,"samples":179734},{"bound_ms":5,"rps":{"value":30107,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1644,"first_disqualified_conc":128,"lower_bound":false,"samples":180664},{"bound_ms":10,"rps":{"value":30107,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1644,"first_disqualified_conc":256,"lower_bound":false,"samples":180664},{"bound_ms":50,"rps":{"value":30107,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1644,"first_disqualified_conc":1024,"lower_bound":false,"samples":180664},{"bound_ms":100,"rps":{"value":30107,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1644,"first_disqualified_conc":2048,"lower_bound":false,"samples":180664},{"bound_ms":null,"rps":{"value":30107,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":1644,"first_disqualified_conc":16384,"lower_bound":false,"samples":180664}],"sample_counts":{"gateway_c1_samples":34929,"direct_c1_samples":262960},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f:cohere:gemini","gateway_id":"busbar","gateway_build":"getbusbar/busbar:1.5.5@sha256:b4cfbc7431946204aef5b1f627616e33c654a4b1e12a13758b62a9e6f71501d3","run_id":"run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","measured_at":"2026-11-10T17:39:31Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.958450711545858,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":99045,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":100,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":112,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":139,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":88.31158922713054,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":11323.54211663067,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9612572197789612,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":262140,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":43684,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":10,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":7.421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":20.2578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":15.55078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":20.90625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":20.58984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.7100808639083078,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.004007841246125101,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":46982,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":521,"first_disqualified_conc":32,"lower_bound":false,"samples":281924},{"bound_ms":5,"rps":{"value":47826,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2093,"first_disqualified_conc":256,"lower_bound":false,"samples":286965},{"bound_ms":10,"rps":{"value":47826,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2093,"first_disqualified_conc":256,"lower_bound":false,"samples":286965},{"bound_ms":50,"rps":{"value":47826,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2093,"first_disqualified_conc":1024,"lower_bound":false,"samples":286965},{"bound_ms":100,"rps":{"value":47826,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2093,"first_disqualified_conc":2048,"lower_bound":false,"samples":286965},{"bound_ms":null,"rps":{"value":47826,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":2093,"first_disqualified_conc":16384,"lower_bound":false,"samples":286965}],"sample_counts":{"gateway_c1_samples":49012,"direct_c1_samples":269799},"provenance":{"source_url":"https://onthebench.ai/gateways/busbar","raw_result_url":"https://onthebench.ai/api/v1/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-busbar-16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"16d2da7bc28375e55402b153f0f49b1550d5556f6ebb71eda60b8c3c47ad8d5f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:cohere:anthropic","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:openai:anthropic","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":332,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":508,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":538,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":424.9108011676938,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":2353.4351145038167,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7074297904213582,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":40079,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":6679,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":482,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":635,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":21,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":47311.46651648458,"certified":true,"suppressed":false,"headroom":0.8170565763537634,"rig_ceiling":57904.7619047619,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1140,"certified":true,"suppressed":false,"headroom":0.8170565763537634,"rig_ceiling":57904.7619047619,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":52.9140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":87.58203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":87.0859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":89.44140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":89.0703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.11046857868890064,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.3226255858418706,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":4631,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":894,"first_disqualified_conc":4,"lower_bound":false,"samples":27792},{"bound_ms":5,"rps":{"value":6656,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4122,"first_disqualified_conc":16,"lower_bound":false,"samples":39947},{"bound_ms":10,"rps":{"value":6820,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":9023,"first_disqualified_conc":32,"lower_bound":false,"samples":40929},{"bound_ms":50,"rps":{"value":7453,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":33436,"first_disqualified_conc":128,"lower_bound":false,"samples":44724},{"bound_ms":100,"rps":{"value":7474,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":66381,"first_disqualified_conc":256,"lower_bound":false,"samples":44854},{"bound_ms":null,"rps":{"value":7474,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":66381,"first_disqualified_conc":null,"lower_bound":false,"samples":44854}],"sample_counts":{"gateway_c1_samples":16334,"direct_c1_samples":245181,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1280,"http_status_counts":{"200":1280},"stream_errors_no_response":0},{"conc":1152,"http_status_counts":{"200":1152},"stream_errors_no_response":0},{"conc":1088,"http_status_counts":{"200":1088},"stream_errors_no_response":0},{"conc":1120,"http_status_counts":{"200":1120},"stream_errors_no_response":0},{"conc":1136,"http_status_counts":{"200":1136},"stream_errors_no_response":0},{"conc":1144,"http_status_counts":{"200":1144},"stream_errors_no_response":0},{"conc":1140,"http_status_counts":{"200":1140},"stream_errors_no_response":0},{"conc":1142,"http_status_counts":{"200":1142},"stream_errors_no_response":0},{"conc":1141,"http_status_counts":{"200":1141},"stream_errors_no_response":0},{"conc":1140,"http_status_counts":{"200":1140},"stream_errors_no_response":0},{"conc":1140,"http_status_counts":{"200":1140},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:openai-responses:anthropic","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":363,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":639,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":669,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":571.993208450469,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1748.272506082725,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.8537078031502604,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":35927,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":5987,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":12,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.009352297714810588,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":53.15625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":240.859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":157.453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":324.51953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":324.09375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":10.70424987337067,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.35096701211601694,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":4280,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":991,"first_disqualified_conc":4,"lower_bound":false,"samples":25689},{"bound_ms":5,"rps":{"value":5982,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4904,"first_disqualified_conc":16,"lower_bound":false,"samples":35897},{"bound_ms":10,"rps":{"value":5982,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4904,"first_disqualified_conc":16,"lower_bound":false,"samples":35897},{"bound_ms":50,"rps":{"value":6343,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":42529,"first_disqualified_conc":128,"lower_bound":false,"samples":38060},{"bound_ms":100,"rps":{"value":6343,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":42529,"first_disqualified_conc":256,"lower_bound":false,"samples":38060},{"bound_ms":null,"rps":{"value":6343,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":42529,"first_disqualified_conc":null,"lower_bound":false,"samples":38060}],"sample_counts":{"gateway_c1_samples":14995,"direct_c1_samples":248903},"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:bedrock:anthropic","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:gemini:anthropic","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:anthropic:anthropic","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":386,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":722,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":753,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":31,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":498.8399071925754,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":2004.6511627906975,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.785925521214345,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":37928,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":6320,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":485,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":1078,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21069us) came in under the mock's (21073us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21128us) came in under the mock's (21135us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":64695.44352492808,"certified":true,"suppressed":false,"headroom":0.8169926519544719,"rig_ceiling":79187.30158730158,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1559,"certified":true,"suppressed":false,"headroom":0.8169926519544719,"rig_ceiling":79187.30158730158,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":52.9609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":86.8515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":87.08203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":88.515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":88.10546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.2543254131118377,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.3148156669010287,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":2351,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":736,"first_disqualified_conc":2,"lower_bound":false,"samples":14110},{"bound_ms":5,"rps":{"value":6462,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4288,"first_disqualified_conc":16,"lower_bound":false,"samples":38778},{"bound_ms":10,"rps":{"value":6654,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":9045,"first_disqualified_conc":32,"lower_bound":false,"samples":39931},{"bound_ms":50,"rps":{"value":7222,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":32554,"first_disqualified_conc":128,"lower_bound":false,"samples":43337},{"bound_ms":100,"rps":{"value":7237,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":64064,"first_disqualified_conc":256,"lower_bound":false,"samples":43430},{"bound_ms":null,"rps":{"value":7237,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":64064,"first_disqualified_conc":null,"lower_bound":false,"samples":43430}],"sample_counts":{"gateway_c1_samples":14086,"direct_c1_samples":240685,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1792,"http_status_counts":{"200":1792},"stream_errors_no_response":0},{"conc":1664,"http_status_counts":{"200":1664},"stream_errors_no_response":0},{"conc":1600,"http_status_counts":{"200":1600},"stream_errors_no_response":0},{"conc":1568,"http_status_counts":{"200":1568},"stream_errors_no_response":0},{"conc":1552,"http_status_counts":{"200":1552},"stream_errors_no_response":0},{"conc":1560,"http_status_counts":{"200":1560},"stream_errors_no_response":0},{"conc":1556,"http_status_counts":{"200":1556},"stream_errors_no_response":0},{"conc":1558,"http_status_counts":{"200":1558},"stream_errors_no_response":0},{"conc":1559,"http_status_counts":{"200":1559},"stream_errors_no_response":0},{"conc":1559,"http_status_counts":{"200":1559},"stream_errors_no_response":0},{"conc":1559,"http_status_counts":{"200":1559},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:bedrock:openai","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:cohere:openai","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:openai-responses:openai","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:gemini:openai","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:anthropic:openai","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":348,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":531,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":561,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":449.57994470837696,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":2224.2985074626868,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.695593380601896,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":37257,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":6208,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.0055291623050701885,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":482,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":681,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21066us) came in under the mock's (21073us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":61069.630667374906,"certified":true,"suppressed":false,"headroom":0.782752834481734,"rig_ceiling":78019.04761904762,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1536,"certified":true,"suppressed":false,"headroom":0.782752834481734,"rig_ceiling":78019.04761904762,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":53.23828125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":87.93359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":89.765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":90.3125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":90,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.027332179074735487,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.19131230185335213,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":4318,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":909,"first_disqualified_conc":4,"lower_bound":false,"samples":25914},{"bound_ms":5,"rps":{"value":6214,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4547,"first_disqualified_conc":16,"lower_bound":false,"samples":37292},{"bound_ms":10,"rps":{"value":6384,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":9602,"first_disqualified_conc":32,"lower_bound":false,"samples":38313},{"bound_ms":50,"rps":{"value":6988,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":35107,"first_disqualified_conc":128,"lower_bound":false,"samples":41937},{"bound_ms":100,"rps":{"value":7071,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":68653,"first_disqualified_conc":256,"lower_bound":false,"samples":42435},{"bound_ms":null,"rps":{"value":7071,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":68653,"first_disqualified_conc":null,"lower_bound":false,"samples":42435}],"sample_counts":{"gateway_c1_samples":15595,"direct_c1_samples":242839,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1792,"http_status_counts":{"200":1792},"stream_errors_no_response":0},{"conc":1664,"http_status_counts":{"200":1664},"stream_errors_no_response":0},{"conc":1600,"http_status_counts":{"200":1600},"stream_errors_no_response":0},{"conc":1568,"http_status_counts":{"200":1568},"stream_errors_no_response":0},{"conc":1552,"http_status_counts":{"200":1552},"stream_errors_no_response":0},{"conc":1544,"http_status_counts":{"200":1544},"stream_errors_no_response":0},{"conc":1540,"http_status_counts":{"200":1540},"stream_errors_no_response":0},{"conc":1538,"http_status_counts":{"200":1538},"stream_errors_no_response":0},{"conc":1537,"http_status_counts":{"200":1537},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:openai:openai","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":343,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":565,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":596,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":31,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":439.1503284680862,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":2277.1245634458674,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7135981462725213,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":39121,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":6519,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":479,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":601,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21067us) came in under the mock's (21074us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":26,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":50390.08056872798,"certified":true,"suppressed":false,"headroom":0.8212373437059869,"rig_ceiling":61358.730158730155,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1208,"certified":true,"suppressed":false,"headroom":0.8212373437059869,"rig_ceiling":61358.730158730155,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":53.00390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":87.26171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":90.45703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":90.5078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":89.015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.33822817505778513,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.41083926712820346,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":4555,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":897,"first_disqualified_conc":4,"lower_bound":false,"samples":27335},{"bound_ms":5,"rps":{"value":6685,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4002,"first_disqualified_conc":16,"lower_bound":false,"samples":40117},{"bound_ms":10,"rps":{"value":6862,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":8817,"first_disqualified_conc":32,"lower_bound":false,"samples":41180},{"bound_ms":50,"rps":{"value":7415,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":33837,"first_disqualified_conc":128,"lower_bound":false,"samples":44496},{"bound_ms":100,"rps":{"value":7475,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":66788,"first_disqualified_conc":256,"lower_bound":false,"samples":44859},{"bound_ms":null,"rps":{"value":7475,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":66788,"first_disqualified_conc":null,"lower_bound":false,"samples":44859}],"sample_counts":{"gateway_c1_samples":15837,"direct_c1_samples":239533,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1280,"http_status_counts":{"200":1280},"stream_errors_no_response":0},{"conc":1152,"http_status_counts":{"200":1152},"stream_errors_no_response":0},{"conc":1216,"http_status_counts":{"200":1216},"stream_errors_no_response":0},{"conc":1184,"http_status_counts":{"200":1184},"stream_errors_no_response":0},{"conc":1200,"http_status_counts":{"200":1200},"stream_errors_no_response":0},{"conc":1208,"http_status_counts":{"200":1208},"stream_errors_no_response":0},{"conc":1212,"http_status_counts":{"200":1212},"stream_errors_no_response":0},{"conc":1210,"http_status_counts":{"200":1210},"stream_errors_no_response":0},{"conc":1209,"http_status_counts":{"200":1209},"stream_errors_no_response":0},{"conc":1208,"http_status_counts":{"200":1208},"stream_errors_no_response":0},{"conc":1208,"http_status_counts":{"200":1208},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:gemini:cohere","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:openai-responses:cohere","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":353,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":600,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":630,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":569.9209901099508,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1754.6291808046535,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.8569936858107657,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":36198,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":6032,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":12,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.013011768606000331,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":53.12109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":229.89453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":145.29296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":316.98046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":316.4453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still falling - releasing memory, not leaking it when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-59.167562410516275,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.3443264303539636,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":4322,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":972,"first_disqualified_conc":4,"lower_bound":false,"samples":25940},{"bound_ms":5,"rps":{"value":6006,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4878,"first_disqualified_conc":16,"lower_bound":false,"samples":36046},{"bound_ms":10,"rps":{"value":6006,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4878,"first_disqualified_conc":16,"lower_bound":false,"samples":36046},{"bound_ms":50,"rps":{"value":6311,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":41872,"first_disqualified_conc":128,"lower_bound":false,"samples":37871},{"bound_ms":100,"rps":{"value":6311,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":41872,"first_disqualified_conc":256,"lower_bound":false,"samples":37871},{"bound_ms":null,"rps":{"value":6311,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":41872,"first_disqualified_conc":null,"lower_bound":false,"samples":37871}],"sample_counts":{"gateway_c1_samples":15319,"direct_c1_samples":247454},"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:anthropic:cohere","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":329,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":513,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":543,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":426.17841838722245,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":2346.435100548446,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.6817240720590323,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":38505,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":6416,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":12,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.005609661082976237,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":53.11328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":87.6015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":88.75390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":89.65234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":89.16015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.11396281504661937,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.2512557476501924,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":4543,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":888,"first_disqualified_conc":4,"lower_bound":false,"samples":27264},{"bound_ms":5,"rps":{"value":6320,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4498,"first_disqualified_conc":16,"lower_bound":false,"samples":37931},{"bound_ms":10,"rps":{"value":6500,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":9463,"first_disqualified_conc":32,"lower_bound":false,"samples":39010},{"bound_ms":50,"rps":{"value":7107,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":34456,"first_disqualified_conc":128,"lower_bound":false,"samples":42647},{"bound_ms":100,"rps":{"value":7172,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":65971,"first_disqualified_conc":256,"lower_bound":false,"samples":43039},{"bound_ms":null,"rps":{"value":7172,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":65971,"first_disqualified_conc":null,"lower_bound":false,"samples":43039}],"sample_counts":{"gateway_c1_samples":16446,"direct_c1_samples":244026},"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:cohere:cohere","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:bedrock:cohere","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:openai:cohere","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":335,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":515,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":546,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":31,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":418.7076923076923,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":2388.3009994121103,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7066179481631804,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":40625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":6769,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":12,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.005267692307692307,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":52.87890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":87.42578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":90.28125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":90.28125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":88.984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.20381370094580964,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.25987586900840676,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":4690,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":831,"first_disqualified_conc":4,"lower_bound":false,"samples":28153},{"bound_ms":5,"rps":{"value":6742,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":3930,"first_disqualified_conc":16,"lower_bound":false,"samples":40458},{"bound_ms":10,"rps":{"value":6886,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":8796,"first_disqualified_conc":32,"lower_bound":false,"samples":41331},{"bound_ms":50,"rps":{"value":7538,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":32969,"first_disqualified_conc":128,"lower_bound":false,"samples":45232},{"bound_ms":100,"rps":{"value":7549,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":64103,"first_disqualified_conc":256,"lower_bound":false,"samples":45303},{"bound_ms":null,"rps":{"value":7549,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":64103,"first_disqualified_conc":null,"lower_bound":false,"samples":45303}],"sample_counts":{"gateway_c1_samples":16166,"direct_c1_samples":236668},"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:anthropic:bedrock","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":452,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":774,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":804,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":553.5889579676949,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1806.3944116066632,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7731420943510743,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":33617,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":5602,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":12,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.006990510753487818,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":52.88671875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":91.16796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":91.66796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":93.25390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":92.7109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.046980188503311364,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.38419409473514465,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":2020,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":627,"first_disqualified_conc":2,"lower_bound":false,"samples":12127},{"bound_ms":5,"rps":{"value":5567,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4918,"first_disqualified_conc":16,"lower_bound":false,"samples":33409},{"bound_ms":10,"rps":{"value":5567,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4918,"first_disqualified_conc":16,"lower_bound":false,"samples":33409},{"bound_ms":50,"rps":{"value":6093,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":41628,"first_disqualified_conc":128,"lower_bound":false,"samples":36562},{"bound_ms":100,"rps":{"value":6134,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":82421,"first_disqualified_conc":256,"lower_bound":false,"samples":36811},{"bound_ms":null,"rps":{"value":6134,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":82421,"first_disqualified_conc":null,"lower_bound":false,"samples":36811}],"sample_counts":{"gateway_c1_samples":12219,"direct_c1_samples":247325},"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:cohere:bedrock","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:gemini:bedrock","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:bedrock:bedrock","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:openai-responses:bedrock","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":471,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":809,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":839,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":674.6747391472369,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1482.1957040572793,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.870399681280061,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":31052,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":5174,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.023605564858946285,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":53.0703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":193.3359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":130.703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":276.3125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":275.9609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still falling - releasing memory, not leaking it when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-105.13461029863961,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.27690046602973994,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":1945,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":799,"first_disqualified_conc":2,"lower_bound":false,"samples":11672},{"bound_ms":5,"rps":{"value":4809,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":2400,"first_disqualified_conc":8,"lower_bound":false,"samples":28860},{"bound_ms":10,"rps":{"value":5149,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":5562,"first_disqualified_conc":16,"lower_bound":false,"samples":30902},{"bound_ms":50,"rps":{"value":5409,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":46957,"first_disqualified_conc":128,"lower_bound":false,"samples":32459},{"bound_ms":100,"rps":{"value":5409,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":46957,"first_disqualified_conc":256,"lower_bound":false,"samples":32459},{"bound_ms":null,"rps":{"value":5409,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":46957,"first_disqualified_conc":null,"lower_bound":false,"samples":32459}],"sample_counts":{"gateway_c1_samples":11671,"direct_c1_samples":247559},"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:openai:bedrock","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":450,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":756,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":787,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":31,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":545.4287739192063,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1833.4199584199587,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7992734367170747,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":35275,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":5878,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":12,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":52.40234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":91.28515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":88.5234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":92.87890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":92.6640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.28900171285631676,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.29023336794601473,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":2045,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":694,"first_disqualified_conc":2,"lower_bound":false,"samples":12274},{"bound_ms":5,"rps":{"value":5862,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4478,"first_disqualified_conc":16,"lower_bound":false,"samples":35177},{"bound_ms":10,"rps":{"value":5862,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4478,"first_disqualified_conc":16,"lower_bound":false,"samples":35177},{"bound_ms":50,"rps":{"value":6347,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":40996,"first_disqualified_conc":128,"lower_bound":false,"samples":38087},{"bound_ms":100,"rps":{"value":6361,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":81120,"first_disqualified_conc":256,"lower_bound":false,"samples":38177},{"bound_ms":null,"rps":{"value":6361,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":81120,"first_disqualified_conc":null,"lower_bound":false,"samples":38177}],"sample_counts":{"gateway_c1_samples":12265,"direct_c1_samples":239336},"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:cohere:gemini","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:bedrock:gemini","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:gemini:gemini","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:anthropic:gemini","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":336,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":499,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":530,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":31,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":432.495001578449,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":2312.1654501216544,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.68298447343346,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":38012,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":6334,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":12,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":53.17578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":88.13671875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":88.40625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":89.99609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":89.93359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.2500767982750217,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.5579577971970239,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":4493,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":910,"first_disqualified_conc":4,"lower_bound":false,"samples":26962},{"bound_ms":5,"rps":{"value":6289,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4596,"first_disqualified_conc":16,"lower_bound":false,"samples":37743},{"bound_ms":10,"rps":{"value":6450,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":9533,"first_disqualified_conc":32,"lower_bound":false,"samples":38708},{"bound_ms":50,"rps":{"value":7092,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":33829,"first_disqualified_conc":128,"lower_bound":false,"samples":42558},{"bound_ms":100,"rps":{"value":7137,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":66709,"first_disqualified_conc":256,"lower_bound":false,"samples":42837},{"bound_ms":null,"rps":{"value":7137,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":66709,"first_disqualified_conc":null,"lower_bound":false,"samples":42837}],"sample_counts":{"gateway_c1_samples":16147,"direct_c1_samples":237693},"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:openai-responses:gemini","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":353,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":599,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":629,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":571.7538598245029,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1749.0043710539096,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.8553760435685774,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":36012,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":6001,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":12,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.01721648339442408,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":53.44921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":280.70703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":149.56640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":371.6796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":371.296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":70.65268746342376,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.6650826205323993,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":4292,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":974,"first_disqualified_conc":4,"lower_bound":false,"samples":25758},{"bound_ms":5,"rps":{"value":5944,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4888,"first_disqualified_conc":16,"lower_bound":false,"samples":35670},{"bound_ms":10,"rps":{"value":5944,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4888,"first_disqualified_conc":16,"lower_bound":false,"samples":35670},{"bound_ms":50,"rps":{"value":6212,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":43268,"first_disqualified_conc":128,"lower_bound":false,"samples":37277},{"bound_ms":100,"rps":{"value":6215,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":79641,"first_disqualified_conc":256,"lower_bound":false,"samples":37303},{"bound_ms":null,"rps":{"value":6215,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":79641,"first_disqualified_conc":null,"lower_bound":false,"samples":37303}],"sample_counts":{"gateway_c1_samples":15337,"direct_c1_samples":250993},"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:openai:gemini","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":336,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":508,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":539,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":31,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":424.58713765404383,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":2355.229142185664,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7070624545761913,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":40086,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":6680,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":12,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":53.0625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":87.9375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":91.47265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":91.51171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":89.37890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.08107558270350812,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.4856319041520852,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":4561,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":801,"first_disqualified_conc":4,"lower_bound":false,"samples":27372},{"bound_ms":5,"rps":{"value":6538,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4293,"first_disqualified_conc":16,"lower_bound":false,"samples":39236},{"bound_ms":10,"rps":{"value":6721,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":9630,"first_disqualified_conc":32,"lower_bound":false,"samples":40333},{"bound_ms":50,"rps":{"value":7332,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":35530,"first_disqualified_conc":128,"lower_bound":false,"samples":43996},{"bound_ms":100,"rps":{"value":7393,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":69364,"first_disqualified_conc":256,"lower_bound":false,"samples":44369},{"bound_ms":null,"rps":{"value":7393,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":69364,"first_disqualified_conc":null,"lower_bound":false,"samples":44369}],"sample_counts":{"gateway_c1_samples":16158,"direct_c1_samples":241090},"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:anthropic:openai-responses","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:openai:openai-responses","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:openai-responses:openai-responses","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":363,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":591,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":621,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":577.7154870357393,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1730.9558466763708,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.8561883515203386,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":35675,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":5945,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":12,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":52.81640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":224.859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":125.69921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":326.2734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":325.96875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still climbing when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":27.225418082714604,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.30918765034361817,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":2485,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":602,"first_disqualified_conc":2,"lower_bound":false,"samples":14918},{"bound_ms":5,"rps":{"value":5918,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4963,"first_disqualified_conc":16,"lower_bound":false,"samples":35519},{"bound_ms":10,"rps":{"value":5918,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4963,"first_disqualified_conc":16,"lower_bound":false,"samples":35519},{"bound_ms":50,"rps":{"value":6295,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":41234,"first_disqualified_conc":128,"lower_bound":false,"samples":37775},{"bound_ms":100,"rps":{"value":6295,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":41234,"first_disqualified_conc":256,"lower_bound":false,"samples":37775},{"bound_ms":null,"rps":{"value":6295,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":41234,"first_disqualified_conc":null,"lower_bound":false,"samples":37775}],"sample_counts":{"gateway_c1_samples":15015,"direct_c1_samples":246146},"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:gemini:openai-responses","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:cohere:openai-responses","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad:bedrock:openai-responses","gateway_id":"gomodel","gateway_build":"enterpilot/gomodel:0.2.3@sha256:c50627e093ff88bff311e582a43d7de14e0be29d52cb333ecb3191eac7874f05","run_id":"run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","measured_at":"2026-11-10T17:39:52Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","detail":"GoModel v0.2.3 source 86dbeffc254e0166367d72e0404f5a27aceeeaaa: local single-request checks using the engine's request bodies and authentication headers establish 16 exact protocol pairs; 15 are enabled on the pinned image. The 0.2.3 default configured-model allowlist remains enabled; no legacy fallback override is applied. OpenAI CompatibleProvider uses Chat Completions for chat/messages ingress and native Responses for responses ingress; the other OpenAI cross-operation cells do not select the requested upstream wire. Cohere supports translated chat/responses/messages. Native Cohere passthrough is opt-in (not in the default ENABLED_PASSTHROUGH_PROVIDERS), so cohere>cohere is not configured under the benchmark's out-of-the-box rule. Gemini and Bedrock implement upstream adapters but no native ingress/passthrough interface. These checks verify protocol routing only, not performance. Evidence: results/diagnostics/refresh-configuration/gomodel-0.2.3-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.290044595058161,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98402,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/gomodel","raw_result_url":"https://onthebench.ai/api/v1/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-gomodel-f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"f6e038780b29f86549f840f902ac96740154f0f0ef2a66a3fd9dcd4bc3b83bad"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:bedrock:cohere","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:anthropic:cohere","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:gemini:cohere","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:cohere:cohere","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:openai:cohere","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:openai-responses:cohere","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:anthropic:openai","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:openai-responses:openai","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:openai:openai","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{"added_latency_p50_us":{"value":256,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":274,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":259.381777553837,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":3855.320945945946,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9840922276022175,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":91294,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":15213,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":24,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":399,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":493,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21132us) came in under the mock's (21146us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":21883.075061106818,"certified":true,"suppressed":false,"headroom":0.9305033267074309,"rig_ceiling":23517.460317460318,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":463,"certified":true,"suppressed":false,"headroom":0.9305033267074309,"rig_ceiling":23517.460317460318,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":42.97265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":56.1796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":53.69140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":56.94921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":56.609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":126,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.4999152222931118,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.12448848274528536,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":15405,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":750,"first_disqualified_conc":16,"lower_bound":false,"samples":92444},{"bound_ms":5,"rps":{"value":16288,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":3076,"first_disqualified_conc":64,"lower_bound":false,"samples":97741},{"bound_ms":10,"rps":{"value":16376,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":6295,"first_disqualified_conc":128,"lower_bound":false,"samples":98266},{"bound_ms":50,"rps":{"value":16376,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":6295,"first_disqualified_conc":128,"lower_bound":false,"samples":98266},{"bound_ms":100,"rps":{"value":16376,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":6295,"first_disqualified_conc":128,"lower_bound":false,"samples":98266},{"bound_ms":null,"rps":{"value":16376,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":6295,"first_disqualified_conc":128,"lower_bound":false,"samples":98266}],"sample_counts":{"gateway_c1_samples":21418,"direct_c1_samples":260967,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":460,"404":52},"stream_errors_no_response":0},{"conc":384,"http_status_counts":{"200":384},"stream_errors_no_response":0},{"conc":448,"http_status_counts":{"200":448},"stream_errors_no_response":0},{"conc":480,"http_status_counts":{"200":391,"404":89},"stream_errors_no_response":0},{"conc":464,"http_status_counts":{"200":463,"404":1},"stream_errors_no_response":0},{"conc":456,"http_status_counts":{"200":456},"stream_errors_no_response":0},{"conc":460,"http_status_counts":{"200":460},"stream_errors_no_response":0},{"conc":462,"http_status_counts":{"200":462},"stream_errors_no_response":0},{"conc":463,"http_status_counts":{"200":463},"stream_errors_no_response":0},{"conc":463,"http_status_counts":{"200":463},"stream_errors_no_response":0},{"conc":463,"http_status_counts":{"200":463},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:cohere:openai","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:bedrock:openai","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:gemini:openai","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:openai-responses:anthropic","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:cohere:anthropic","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:openai:anthropic","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{"added_latency_p50_us":{"value":259,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":278,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":305,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":260.87723553928595,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":3833.2206255283177,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9833009260344443,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":90694,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":15114,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":26,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":400,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":515,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":15,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":20,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":22502.421498049265,"certified":true,"suppressed":false,"headroom":0.9287556042826938,"rig_ceiling":24228.571428571428,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":477,"certified":true,"suppressed":false,"headroom":0.9287556042826938,"rig_ceiling":24228.571428571428,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":42.8515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":56.48046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":53.44140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":57.0390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":56.78515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":168,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.10989598445016407,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.1243941906234104,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":15212,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":767,"first_disqualified_conc":16,"lower_bound":false,"samples":91291},{"bound_ms":5,"rps":{"value":16081,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":3103,"first_disqualified_conc":64,"lower_bound":false,"samples":96498},{"bound_ms":10,"rps":{"value":16164,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":6403,"first_disqualified_conc":128,"lower_bound":false,"samples":97003},{"bound_ms":50,"rps":{"value":16164,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":6403,"first_disqualified_conc":128,"lower_bound":false,"samples":97003},{"bound_ms":100,"rps":{"value":16164,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":6403,"first_disqualified_conc":128,"lower_bound":false,"samples":97003},{"bound_ms":null,"rps":{"value":16164,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":6403,"first_disqualified_conc":128,"lower_bound":false,"samples":97003}],"sample_counts":{"gateway_c1_samples":21200,"direct_c1_samples":264063,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":511,"404":1},"stream_errors_no_response":0},{"conc":384,"http_status_counts":{"200":384},"stream_errors_no_response":0},{"conc":448,"http_status_counts":{"200":448},"stream_errors_no_response":0},{"conc":480,"http_status_counts":{"200":463,"404":17},"stream_errors_no_response":0},{"conc":464,"http_status_counts":{"200":464},"stream_errors_no_response":0},{"conc":472,"http_status_counts":{"200":472},"stream_errors_no_response":0},{"conc":476,"http_status_counts":{"200":476},"stream_errors_no_response":0},{"conc":478,"http_status_counts":{"200":378,"404":100},"stream_errors_no_response":0},{"conc":477,"http_status_counts":{"200":477},"stream_errors_no_response":0},{"conc":477,"http_status_counts":{"200":477},"stream_errors_no_response":0},{"conc":477,"http_status_counts":{"200":477},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:bedrock:anthropic","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:anthropic:anthropic","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{"added_latency_p50_us":{"value":175,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":192,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":220,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":166.79831861652607,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5995.264270613108,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9827635256920014,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":141788,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":23628,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":270,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":404,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21067us) came in under the mock's (21069us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21141us) came in under the mock's (21159us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":51526.561485962746,"certified":true,"suppressed":false,"headroom":0.8882917506610258,"rig_ceiling":58006.34920634921,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1142,"certified":true,"suppressed":false,"headroom":0.8882917506610258,"rig_ceiling":58006.34920634921,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":42.9453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":50.80078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":50.78515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":51.328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":51.1328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.21467345625353568,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.12427634769142537,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":24452,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":984,"first_disqualified_conc":32,"lower_bound":false,"samples":146729},{"bound_ms":5,"rps":{"value":25909,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":3987,"first_disqualified_conc":128,"lower_bound":false,"samples":155481},{"bound_ms":10,"rps":{"value":26217,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":8810,"first_disqualified_conc":256,"lower_bound":false,"samples":157328},{"bound_ms":50,"rps":{"value":26320,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":19321,"first_disqualified_conc":1024,"lower_bound":false,"samples":157941},{"bound_ms":100,"rps":{"value":26320,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":19321,"first_disqualified_conc":2048,"lower_bound":false,"samples":157941},{"bound_ms":null,"rps":{"value":26320,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":19321,"first_disqualified_conc":null,"lower_bound":false,"samples":157941}],"sample_counts":{"gateway_c1_samples":30620,"direct_c1_samples":260783,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1280,"http_status_counts":{"200":1280},"stream_errors_no_response":0},{"conc":1152,"http_status_counts":{"200":1152},"stream_errors_no_response":0},{"conc":1088,"http_status_counts":{"200":1088},"stream_errors_no_response":0},{"conc":1120,"http_status_counts":{"200":1120},"stream_errors_no_response":0},{"conc":1136,"http_status_counts":{"200":1136},"stream_errors_no_response":0},{"conc":1144,"http_status_counts":{"200":1144},"stream_errors_no_response":0},{"conc":1140,"http_status_counts":{"200":1140},"stream_errors_no_response":0},{"conc":1142,"http_status_counts":{"200":1142},"stream_errors_no_response":0},{"conc":1143,"http_status_counts":{"200":1143},"stream_errors_no_response":0},{"conc":1142,"http_status_counts":{"200":1142},"stream_errors_no_response":0},{"conc":1142,"http_status_counts":{"200":1142},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:gemini:anthropic","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:openai:bedrock","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{"added_latency_p50_us":{"value":293,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":311,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":338,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":295.1100305157913,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":3388.5666246321985,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9886809407528605,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":80614,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":13433,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":19,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":42.97265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":56.4921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":53.671875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":56.91015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":56.7265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":138,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.041694531242001856,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.12441485584633757,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":13411,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":855,"first_disqualified_conc":16,"lower_bound":false,"samples":80477},{"bound_ms":5,"rps":{"value":14128,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":3505,"first_disqualified_conc":64,"lower_bound":false,"samples":84779},{"bound_ms":10,"rps":{"value":14164,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":7248,"first_disqualified_conc":128,"lower_bound":false,"samples":84985},{"bound_ms":50,"rps":{"value":14164,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":7248,"first_disqualified_conc":256,"lower_bound":false,"samples":84985},{"bound_ms":100,"rps":{"value":14164,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":7248,"first_disqualified_conc":256,"lower_bound":false,"samples":84985},{"bound_ms":null,"rps":{"value":14164,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":7248,"first_disqualified_conc":256,"lower_bound":false,"samples":84985}],"sample_counts":{"gateway_c1_samples":18960,"direct_c1_samples":263027},"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:gemini:bedrock","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:openai-responses:bedrock","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:cohere:bedrock","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:anthropic:bedrock","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:bedrock:bedrock","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{"added_latency_p50_us":{"value":203,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":220,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":247,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":191.48936170212767,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5222.222222222222,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9872755366427802,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":124080,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":20677,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.000008059316569954868,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":42.9609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":50.83203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":51.14453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":51.4375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":51.44140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.0079210740835309,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.1244287498577021,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":20749,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":540,"first_disqualified_conc":16,"lower_bound":false,"samples":124513},{"bound_ms":5,"rps":{"value":22637,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":4547,"first_disqualified_conc":128,"lower_bound":false,"samples":135825},{"bound_ms":10,"rps":{"value":22807,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":9943,"first_disqualified_conc":256,"lower_bound":false,"samples":136867},{"bound_ms":50,"rps":{"value":22894,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":20380,"first_disqualified_conc":1024,"lower_bound":false,"samples":137389},{"bound_ms":100,"rps":{"value":22894,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":20380,"first_disqualified_conc":2048,"lower_bound":false,"samples":137389},{"bound_ms":null,"rps":{"value":22894,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":20380,"first_disqualified_conc":null,"lower_bound":false,"samples":137389}],"sample_counts":{"gateway_c1_samples":26758,"direct_c1_samples":262304},"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:openai:gemini","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:anthropic:gemini","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:gemini:gemini","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:cohere:gemini","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:bedrock:gemini","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:openai-responses:gemini","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:cohere:openai-responses","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:bedrock:openai-responses","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:anthropic:openai-responses","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:openai:openai-responses","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:openai-responses:openai-responses","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558:gemini:openai-responses","gateway_id":"helicone","gateway_build":"target/release/ai-gateway","run_id":"run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","measured_at":"2026-11-10T17:39:37Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","detail":"Pinned v0.2.0-beta.30 source 081f310be779f27122e6038e81e2359c44db4ecb: ai-gateway/src/endpoints defines OpenAI Chat, Anthropic Messages and Bedrock Converse. Google uses v1beta/openai/chat/completions, not native Gemini. router/router_details.rs exposes OpenAI router ingress plus provider-prefixed direct proxy paths. No Responses, native Gemini or Cohere interface is registered; direct proxy does not imply cross-protocol conversion. Five supported exact-wire pairs pass the 36-cell correctness check. Evidence: results/diagnostics/refresh-configuration/helicone-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.773438393330491,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91607,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"ai-gateway","binary_sha256":"f401a719a6341a95be9d1f0b7a63ce385241583ee632f5724df3bf54b04a22f1","cargo_lock_sha256":"d70600602afa1fa8a054251da84622cd764c920e110bf0e3266f0a562473de88","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"081f310be779f27122e6038e81e2359c44db4ecb","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/helicone","raw_result_url":"https://onthebench.ai/api/v1/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-helicone-4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"4a55ff2dcc509c5e03f85bd92e36b949ac410653128ccc77d60019bb3aaf2558"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:cohere:cohere","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":615,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":747,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":774,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":613.2194813409235,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1630.737493553378,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.8053044884180178,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":31620,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":5269,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":143,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.0076217583807716635,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":1207.94921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1217.2890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1218.421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1226.8046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1324.41015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":60,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.7482121208635604,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":1.0318152244070045,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":3092,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":790,"first_disqualified_conc":4,"lower_bound":false,"samples":18555},{"bound_ms":5,"rps":{"value":5921,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":2939,"first_disqualified_conc":16,"lower_bound":false,"samples":35536},{"bound_ms":10,"rps":{"value":6529,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":9093,"first_disqualified_conc":64,"lower_bound":false,"samples":39189},{"bound_ms":50,"rps":{"value":6633,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":18236,"first_disqualified_conc":256,"lower_bound":false,"samples":39805},{"bound_ms":100,"rps":{"value":6633,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":18236,"first_disqualified_conc":512,"lower_bound":false,"samples":39805},{"bound_ms":null,"rps":{"value":6633,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":18236,"first_disqualified_conc":null,"lower_bound":false,"samples":39805}],"sample_counts":{"gateway_c1_samples":9341,"direct_c1_samples":267984},"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:bedrock:cohere","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:gemini:cohere","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:openai:cohere","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_served","raw_state":"failed","reason":"failed","detail":"probed and answered HTTP 200, but the body is not a usable openai response (see body_snippet)","evidence_scope":"Recorded run evidence","probe_http_status":"200","response_excerpt":"the 2xx body is not a response of the openai dialect with assistant text at choices[].message.content: {\"response_id\":\"x\",\"generation_id\":\"x\",\"text\":\"ok\",\"finish_reason\":\"COMPLETE\",\"meta\":{\"api_version\":{\"version\":\"1\"},\"tokens\":{\"input_tokens\":10,\"output_tokens\":2},\"billed_units\":{\"input_tokens\":10,\"output_tokens\":2}}}"},"test":{"ingress_protocol":"openai","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:anthropic:cohere","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_served","raw_state":"failed","reason":"failed","detail":"probed and answered HTTP 200, but the body is not a usable anthropic response (see body_snippet)","evidence_scope":"Recorded run evidence","probe_http_status":"200","response_excerpt":"the 2xx body is not a response of the anthropic dialect with assistant text at content[type=text].text: {\"id\":\"\",\"type\":\"message\",\"role\":\"assistant\",\"content\":null,\"model\":\"\",\"stop_reason\":null,\"stop_sequence\":null,\"usage\":{},\"error\":null}"},"test":{"ingress_protocol":"anthropic","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:openai-responses:cohere","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:cohere:anthropic","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:gemini:anthropic","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:openai:anthropic","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":859,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1112,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1140,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":882.4409213281484,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1133.2203389830509,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.980004881341195,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":26744,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4456,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":145,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.001121746933891714,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":1095,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":1556,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21173us) came in under the mock's (21174us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":23656.559777267295,"certified":true,"suppressed":false,"headroom":0.08075923714495402,"rig_ceiling":292926.98412698414,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":5767,"certified":true,"suppressed":false,"headroom":0.08075923714495402,"rig_ceiling":292926.98412698414,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":1148.421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1158.875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1159.7734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1167.30859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1264.890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":60,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.7021128067897227,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.8890053062912313,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4788,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":4880,"first_disqualified_conc":32,"lower_bound":false,"samples":28736},{"bound_ms":10,"rps":{"value":4909,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":9079,"first_disqualified_conc":64,"lower_bound":false,"samples":29464},{"bound_ms":50,"rps":{"value":4944,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":19849,"first_disqualified_conc":256,"lower_bound":false,"samples":29673},{"bound_ms":100,"rps":{"value":4944,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":19849,"first_disqualified_conc":512,"lower_bound":false,"samples":29673},{"bound_ms":null,"rps":{"value":4944,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":19849,"first_disqualified_conc":null,"lower_bound":false,"samples":29673}],"sample_counts":{"gateway_c1_samples":6739,"direct_c1_samples":262868,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":4096,"http_status_counts":{"200":4096},"stream_errors_no_response":0},{"conc":8192,"http_status_counts":{"200":8082},"stream_errors_no_response":110},{"conc":6144,"http_status_counts":{"200":6144},"stream_errors_no_response":0},{"conc":7168,"http_status_counts":{"200":7168},"stream_errors_no_response":0},{"conc":7680,"http_status_counts":{"200":7680},"stream_errors_no_response":0},{"conc":7424,"http_status_counts":{"200":7424},"stream_errors_no_response":0},{"conc":7552,"http_status_counts":{"200":7552},"stream_errors_no_response":0},{"conc":7488,"http_status_counts":{"200":7488},"stream_errors_no_response":0},{"conc":7456,"http_status_counts":{"200":7456},"stream_errors_no_response":0},{"conc":7440,"http_status_counts":{"200":7440},"stream_errors_no_response":0},{"conc":7432,"http_status_counts":{"200":7432},"stream_errors_no_response":0},{"conc":7436,"http_status_counts":{"200":7436},"stream_errors_no_response":0},{"conc":7438,"http_status_counts":{"200":7438},"stream_errors_no_response":0},{"conc":7439,"http_status_counts":{"200":7439},"stream_errors_no_response":0},{"conc":7438,"http_status_counts":{"200":7438},"stream_errors_no_response":0},{"conc":7438,"http_status_counts":{"200":7438},"stream_errors_no_response":0},{"conc":5767,"http_status_counts":{"200":5767},"stream_errors_no_response":0},{"conc":5767,"http_status_counts":{"200":5767},"stream_errors_no_response":0},{"conc":5767,"http_status_counts":{"200":5767},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:bedrock:anthropic","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:openai-responses:anthropic","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:anthropic:anthropic","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":680,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":822,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":850,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":674.8307904095445,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1481.852953676158,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9771911881943663,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":34868,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":5810,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":144,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.0020075714121830907,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":822,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":1012,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":35,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":52509.65359935977,"certified":true,"suppressed":false,"headroom":0.06307020958070865,"rig_ceiling":832558.7301587302,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":16391,"certified":true,"suppressed":false,"headroom":0.06307020958070865,"rig_ceiling":832558.7301587302,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":1153.921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1161.83984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1162.21484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1171.8125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1272.52734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":60,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.275204255009843,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.5817656189913647,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":1352,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":957,"first_disqualified_conc":2,"lower_bound":false,"samples":8115},{"bound_ms":5,"rps":{"value":5967,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":4103,"first_disqualified_conc":32,"lower_bound":false,"samples":35809},{"bound_ms":10,"rps":{"value":6123,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":9073,"first_disqualified_conc":64,"lower_bound":false,"samples":36743},{"bound_ms":50,"rps":{"value":6224,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":27064,"first_disqualified_conc":256,"lower_bound":false,"samples":37352},{"bound_ms":100,"rps":{"value":6224,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":27064,"first_disqualified_conc":512,"lower_bound":false,"samples":37352},{"bound_ms":null,"rps":{"value":6224,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":27064,"first_disqualified_conc":null,"lower_bound":false,"samples":37352}],"sample_counts":{"gateway_c1_samples":8481,"direct_c1_samples":262245,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":4096,"http_status_counts":{"200":4096},"stream_errors_no_response":0},{"conc":8192,"http_status_counts":{"200":8192},"stream_errors_no_response":0},{"conc":16384,"http_status_counts":{"200":16384},"stream_errors_no_response":0},{"conc":32768,"http_status_counts":{"200":17771},"stream_errors_no_response":14997},{"conc":24576,"http_status_counts":{"200":17562},"stream_errors_no_response":7014},{"conc":20480,"http_status_counts":{"200":17305},"stream_errors_no_response":3175},{"conc":18432,"http_status_counts":{"200":17810},"stream_errors_no_response":622},{"conc":17408,"http_status_counts":{"200":17408},"stream_errors_no_response":0},{"conc":16896,"http_status_counts":{"200":16896},"stream_errors_no_response":0},{"conc":16640,"http_status_counts":{"200":16640},"stream_errors_no_response":0},{"conc":16512,"http_status_counts":{"200":16512},"stream_errors_no_response":0},{"conc":16448,"http_status_counts":{"200":16448},"stream_errors_no_response":0},{"conc":16416,"http_status_counts":{"200":16416},"stream_errors_no_response":0},{"conc":16400,"http_status_counts":{"200":16400},"stream_errors_no_response":0},{"conc":16392,"http_status_counts":{"200":16392},"stream_errors_no_response":0},{"conc":16388,"http_status_counts":{"200":16388},"stream_errors_no_response":0},{"conc":16390,"http_status_counts":{"200":16390},"stream_errors_no_response":0},{"conc":16391,"http_status_counts":{"200":16391},"stream_errors_no_response":0},{"conc":16391,"http_status_counts":{"200":16391},"stream_errors_no_response":0},{"conc":16391,"http_status_counts":{"200":16391},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:openai:openai-responses","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:gemini:openai-responses","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:cohere:openai-responses","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:bedrock:openai-responses","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:openai-responses:openai-responses","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":642,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":747,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":775,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":633.7934164508637,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1577.8011794439765,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9859729286272246,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":37457,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":6242,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":141,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.0010144966227941374,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":1147.16796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1156.55078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1157.92578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1165.46875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1186.5546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":66,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.5987504753473016,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.6971452873324683,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":2925,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":890,"first_disqualified_conc":4,"lower_bound":false,"samples":17553},{"bound_ms":5,"rps":{"value":6438,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":3521,"first_disqualified_conc":32,"lower_bound":false,"samples":38636},{"bound_ms":10,"rps":{"value":6717,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":6634,"first_disqualified_conc":64,"lower_bound":false,"samples":40311},{"bound_ms":50,"rps":{"value":6791,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":26658,"first_disqualified_conc":256,"lower_bound":false,"samples":40754},{"bound_ms":100,"rps":{"value":6791,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":26658,"first_disqualified_conc":512,"lower_bound":false,"samples":40754},{"bound_ms":null,"rps":{"value":6791,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":26658,"first_disqualified_conc":null,"lower_bound":false,"samples":40754}],"sample_counts":{"gateway_c1_samples":8971,"direct_c1_samples":260207},"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:anthropic:openai-responses","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:openai:gemini","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":923,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1226,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1254,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":951.6544794570992,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1050.8015478164732,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7511344229974055,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":19009,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":3167,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":141,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.011047398600662844,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":1150.5859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1160.25390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1160.609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1170.90234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1268.890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":60,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.25904735694337694,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.6054154815784856,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4171,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4447,"first_disqualified_conc":16,"lower_bound":false,"samples":25035},{"bound_ms":10,"rps":{"value":4327,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":6571,"first_disqualified_conc":32,"lower_bound":false,"samples":25969},{"bound_ms":50,"rps":{"value":4449,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":22764,"first_disqualified_conc":256,"lower_bound":false,"samples":26699},{"bound_ms":100,"rps":{"value":4449,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":22764,"first_disqualified_conc":512,"lower_bound":false,"samples":26699},{"bound_ms":null,"rps":{"value":4449,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":22764,"first_disqualified_conc":null,"lower_bound":false,"samples":26699}],"sample_counts":{"gateway_c1_samples":6239,"direct_c1_samples":265561},"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:gemini:gemini","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":750,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":932,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":959,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":737.715063745513,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1355.5369127516778,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7426132259918599,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":24237,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4038,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":140,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.007014069398027809,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":1156.921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1165.796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1166.64453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1175.75390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1269.25,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":66,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":1.08419810667365,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.8945988670581346,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":5241,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":4564,"first_disqualified_conc":32,"lower_bound":false,"samples":31452},{"bound_ms":10,"rps":{"value":5428,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":8261,"first_disqualified_conc":64,"lower_bound":false,"samples":32577},{"bound_ms":50,"rps":{"value":5506,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":16888,"first_disqualified_conc":256,"lower_bound":false,"samples":33045},{"bound_ms":100,"rps":{"value":5506,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":16888,"first_disqualified_conc":512,"lower_bound":false,"samples":33045},{"bound_ms":null,"rps":{"value":5506,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":16888,"first_disqualified_conc":null,"lower_bound":false,"samples":33045}],"sample_counts":{"gateway_c1_samples":7714,"direct_c1_samples":265544},"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:bedrock:gemini","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:cohere:gemini","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:openai-responses:gemini","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:anthropic:gemini","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":1045,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1360,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1388,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1055.3296800424253,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":947.571189279732,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7438065009330738,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":16971,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":2828,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":142,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.01019386011431265,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":1151.53125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1164.61328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1165.828125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1173.3046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1280.23828125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":60,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":1.1668804149685401,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":1.20687057093733,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":3821,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":2813,"first_disqualified_conc":16,"lower_bound":false,"samples":22929},{"bound_ms":10,"rps":{"value":3862,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":8825,"first_disqualified_conc":32,"lower_bound":false,"samples":23185},{"bound_ms":50,"rps":{"value":3961,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":26997,"first_disqualified_conc":256,"lower_bound":false,"samples":23771},{"bound_ms":100,"rps":{"value":3961,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":26997,"first_disqualified_conc":512,"lower_bound":false,"samples":23771},{"bound_ms":null,"rps":{"value":3961,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":26997,"first_disqualified_conc":null,"lower_bound":false,"samples":23771}],"sample_counts":{"gateway_c1_samples":5488,"direct_c1_samples":262925},"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:openai-responses:bedrock","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:gemini:bedrock","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:cohere:bedrock","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:bedrock:bedrock","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":727,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":861,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":889,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":778.1517645882627,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1285.0963597430407,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9697792765122447,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":30007,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":5000,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":143,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.005065484720231946,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":1157.3984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1165.7421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1167.375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1175.8359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1275.18359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":60,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.6896154469149554,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.6918641708843657,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4996,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":3109,"first_disqualified_conc":16,"lower_bound":false,"samples":29985},{"bound_ms":10,"rps":{"value":5179,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":6789,"first_disqualified_conc":32,"lower_bound":false,"samples":31083},{"bound_ms":50,"rps":{"value":5403,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":21391,"first_disqualified_conc":256,"lower_bound":false,"samples":32432},{"bound_ms":100,"rps":{"value":5403,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":21391,"first_disqualified_conc":512,"lower_bound":false,"samples":32432},{"bound_ms":null,"rps":{"value":5403,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":21391,"first_disqualified_conc":null,"lower_bound":false,"samples":32432}],"sample_counts":{"gateway_c1_samples":7954,"direct_c1_samples":263620},"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:openai:bedrock","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":929,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1235,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1263,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":912.3461771232601,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1096.0751796572692,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7513336009437701,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":19828,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":3304,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":144,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.008069396812588258,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":1158.6875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1172.09765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1173.9765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1179.87109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1198.6875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":60,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.9309596103361604,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":1.0752406083257144,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":4316,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":3304,"first_disqualified_conc":16,"lower_bound":false,"samples":25900},{"bound_ms":10,"rps":{"value":4440,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":5213,"first_disqualified_conc":32,"lower_bound":false,"samples":26646},{"bound_ms":50,"rps":{"value":4587,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":20787,"first_disqualified_conc":256,"lower_bound":false,"samples":27530},{"bound_ms":100,"rps":{"value":4587,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":20787,"first_disqualified_conc":512,"lower_bound":false,"samples":27530},{"bound_ms":null,"rps":{"value":4587,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":20787,"first_disqualified_conc":null,"lower_bound":false,"samples":27530}],"sample_counts":{"gateway_c1_samples":6162,"direct_c1_samples":262294},"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:anthropic:bedrock","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:openai:openai","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":718,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":943,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":971,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":709.3196158019531,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1409.801699716714,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7330563381485045,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":24883,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":4146,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":142,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.008399308765020295,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":881,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":1103,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":2,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21141us) came in under the mock's (21153us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":52911.767627714544,"certified":true,"suppressed":false,"headroom":0.617121104958904,"rig_ceiling":85739.68253968254,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1688,"certified":true,"suppressed":false,"headroom":0.617121104958904,"rig_ceiling":85739.68253968254,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":1163.541015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1170.94140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1171.9921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1181.8125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1287.14453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":66,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.8963186649029332,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.6061112080903336,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":2688,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":944,"first_disqualified_conc":4,"lower_bound":false,"samples":16131},{"bound_ms":5,"rps":{"value":5685,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":2498,"first_disqualified_conc":16,"lower_bound":false,"samples":34120},{"bound_ms":10,"rps":{"value":6095,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":8960,"first_disqualified_conc":64,"lower_bound":false,"samples":36576},{"bound_ms":50,"rps":{"value":6138,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":14447,"first_disqualified_conc":256,"lower_bound":false,"samples":36832},{"bound_ms":100,"rps":{"value":6138,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":14447,"first_disqualified_conc":512,"lower_bound":false,"samples":36832},{"bound_ms":null,"rps":{"value":6138,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":14447,"first_disqualified_conc":null,"lower_bound":false,"samples":36832}],"sample_counts":{"gateway_c1_samples":7996,"direct_c1_samples":262593,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1792,"http_status_counts":{"200":1792},"stream_errors_no_response":0},{"conc":1664,"http_status_counts":{"200":1664},"stream_errors_no_response":0},{"conc":1728,"http_status_counts":{"200":1728},"stream_errors_no_response":0},{"conc":1696,"http_status_counts":{"200":1696},"stream_errors_no_response":0},{"conc":1680,"http_status_counts":{"200":1680},"stream_errors_no_response":0},{"conc":1688,"http_status_counts":{"200":1688},"stream_errors_no_response":0},{"conc":1692,"http_status_counts":{"200":1692},"stream_errors_no_response":0},{"conc":1690,"http_status_counts":{"200":1690},"stream_errors_no_response":0},{"conc":1689,"http_status_counts":{"200":1689},"stream_errors_no_response":0},{"conc":1688,"http_status_counts":{"200":1688},"stream_errors_no_response":0},{"conc":1688,"http_status_counts":{"200":1688},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:gemini:openai","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:bedrock:openai","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:cohere:openai","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:anthropic:openai","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":954,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1252,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1280,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":968.311271560369,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1032.7257663628832,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.5011834686494262,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":12465,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":2076,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":142,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.00008022462896109105,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":1262,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":1705,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":163,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":19376.729819513093,"certified":true,"suppressed":false,"headroom":0.07497629094372328,"rig_ceiling":258438.09523809524,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":5088,"certified":true,"suppressed":false,"headroom":0.07497629094372328,"rig_ceiling":258438.09523809524,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":1151.65234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1160.24609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1161.1953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1177.22265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1280.44921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":60,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.5290055438736946,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":1.003040818815334,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":3867,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4852,"first_disqualified_conc":16,"lower_bound":false,"samples":23211},{"bound_ms":10,"rps":{"value":3999,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":5892,"first_disqualified_conc":32,"lower_bound":false,"samples":23998},{"bound_ms":50,"rps":{"value":4118,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":23767,"first_disqualified_conc":256,"lower_bound":false,"samples":24713},{"bound_ms":100,"rps":{"value":4118,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":23767,"first_disqualified_conc":512,"lower_bound":false,"samples":24713},{"bound_ms":null,"rps":{"value":4118,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":23767,"first_disqualified_conc":null,"lower_bound":false,"samples":24713}],"sample_counts":{"gateway_c1_samples":6058,"direct_c1_samples":259722,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":4096,"http_status_counts":{"200":4096},"stream_errors_no_response":0},{"conc":8192,"http_status_counts":{"200":6688},"stream_errors_no_response":1504},{"conc":6144,"http_status_counts":{"200":6144},"stream_errors_no_response":0},{"conc":5120,"http_status_counts":{"200":5120},"stream_errors_no_response":0},{"conc":5632,"http_status_counts":{"200":5632},"stream_errors_no_response":0},{"conc":5888,"http_status_counts":{"200":5888},"stream_errors_no_response":0},{"conc":6016,"http_status_counts":{"200":6016},"stream_errors_no_response":0},{"conc":6080,"http_status_counts":{"200":6080},"stream_errors_no_response":0},{"conc":6112,"http_status_counts":{"200":6112},"stream_errors_no_response":0},{"conc":6096,"http_status_counts":{"200":6096},"stream_errors_no_response":0},{"conc":6088,"http_status_counts":{"200":6088},"stream_errors_no_response":0},{"conc":6084,"http_status_counts":{"200":6084},"stream_errors_no_response":0},{"conc":6082,"http_status_counts":{"200":6082},"stream_errors_no_response":0},{"conc":6081,"http_status_counts":{"200":6081},"stream_errors_no_response":0},{"conc":6080,"http_status_counts":{"200":6080},"stream_errors_no_response":0},{"conc":6080,"http_status_counts":{"200":6080},"stream_errors_no_response":0},{"conc":5088,"http_status_counts":{"200":5088},"stream_errors_no_response":0},{"conc":5088,"http_status_counts":{"200":5088},"stream_errors_no_response":0},{"conc":5088,"http_status_counts":{"200":5088},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270:openai-responses:openai","gateway_id":"higress","gateway_build":"higress-registry.cn-hangzhou.cr.aliyuncs.com/higress/all-in-one:2.2.5@sha256:9846006cb3cf2dafa942deab0e1d004ed30e219fbc9580438f9e3c58e7096422","run_id":"run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","measured_at":"2026-11-10T17:39:42Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","detail":"Higress 2.2.5 with bundled AI Proxy 2.0.3. Source 4deae1dd3b4cb2707273748985e55bef28e747d1 and pinned-image probes establish 13 configured exact-wire pairs; two Cohere translations failed response correctness in the preflight. In the 2026-11-09 run the engine withheld performance and streaming numbers for six configured pairs because re-verification found the request did not arrive on the requested upstream wire: OpenAI to Anthropic, Gemini, Cohere and Bedrock arrived untranslated on the mock's OpenAI Chat Completions endpoint, and Anthropic to Gemini and Anthropic to Cohere also arrived on the OpenAI Chat Completions endpoint. Responses cross-conversion is not implemented; Messages-to-Bedrock uses Anthropic Mantle rather than Converse. Pairs outside the configured 13 are not configured: unsupported translations pass through the original wire or return unsupported API errors. Evidence: results/diagnostics/refresh-configuration/higress-adapter-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":0.6995914718448216,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":96872,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/higress","raw_result_url":"https://onthebench.ai/api/v1/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-higress-558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"558a1502d4a7ea1f2b74c745739723084c953a3ae5dcf16760a2694a92c21270"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:cohere:openai","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:bedrock:openai","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:anthropic:openai","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:openai:openai","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":213,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":410,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":440,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":220.6875669632459,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":4531.292875989446,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.787179337229732,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":85868,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":14309,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.000733684259561187,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":106009,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":106433,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (0us) came in under the mock's (21072us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":168648,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":3731.918948013315,"certified":true,"suppressed":false,"headroom":0.9419506960129761,"rig_ceiling":3961.904761904762,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":78,"certified":true,"suppressed":false,"headroom":0.9419506960129761,"rig_ceiling":3961.904761904762,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":406.2109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":617.4296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":617.546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":617.73828125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":618.64453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":66,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.10537884976821829,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.240647810533821,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":13315,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":719,"first_disqualified_conc":8,"lower_bound":false,"samples":79905},{"bound_ms":5,"rps":{"value":19190,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":1058,"first_disqualified_conc":32,"lower_bound":false,"samples":115157},{"bound_ms":10,"rps":{"value":21287,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":7308,"first_disqualified_conc":128,"lower_bound":false,"samples":127725},{"bound_ms":50,"rps":{"value":22171,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":17439,"first_disqualified_conc":512,"lower_bound":false,"samples":133046},{"bound_ms":100,"rps":{"value":22171,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":17439,"first_disqualified_conc":2048,"lower_bound":false,"samples":133046},{"bound_ms":null,"rps":{"value":22171,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":17439,"first_disqualified_conc":32768,"lower_bound":false,"samples":133046}],"sample_counts":{"gateway_c1_samples":23861,"direct_c1_samples":246038,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":96,"http_status_counts":{"200":96},"stream_errors_no_response":0},{"conc":80,"http_status_counts":{"200":80},"stream_errors_no_response":0},{"conc":72,"http_status_counts":{"200":72},"stream_errors_no_response":0},{"conc":76,"http_status_counts":{"200":76},"stream_errors_no_response":0},{"conc":78,"http_status_counts":{"200":78},"stream_errors_no_response":0},{"conc":79,"http_status_counts":{"200":79},"stream_errors_no_response":0},{"conc":78,"http_status_counts":{"200":78},"stream_errors_no_response":0},{"conc":78,"http_status_counts":{"200":78},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:gemini:openai","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:openai-responses:openai","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:anthropic:anthropic","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:cohere:anthropic","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:openai:anthropic","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":211,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":420,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":450,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":221.73768782291322,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":4509.833262077811,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9716300019201154,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":105485,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":17578,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.001782243920936626,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":84892,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":85699,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (0us) came in under the mock's (21071us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":147628,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":12509.29640828139,"certified":true,"suppressed":false,"headroom":0.9328665645380299,"rig_ceiling":13409.52380952381,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":264,"certified":true,"suppressed":false,"headroom":0.9328665645380299,"rig_ceiling":13409.52380952381,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":406.140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":614.625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":614.74609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":614.87890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":616.01171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":78,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.11096435899246593,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.32907569010107623,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":9859,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":944,"first_disqualified_conc":8,"lower_bound":false,"samples":59165},{"bound_ms":5,"rps":{"value":16975,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":1275,"first_disqualified_conc":32,"lower_bound":false,"samples":101865},{"bound_ms":10,"rps":{"value":18516,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":9382,"first_disqualified_conc":128,"lower_bound":false,"samples":111100},{"bound_ms":50,"rps":{"value":19147,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":17130,"first_disqualified_conc":512,"lower_bound":false,"samples":114896},{"bound_ms":100,"rps":{"value":19147,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":17130,"first_disqualified_conc":1024,"lower_bound":false,"samples":114896},{"bound_ms":null,"rps":{"value":19147,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":17130,"first_disqualified_conc":null,"lower_bound":false,"samples":114896}],"sample_counts":{"gateway_c1_samples":24037,"direct_c1_samples":241342,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":384,"http_status_counts":{"200":384},"stream_errors_no_response":0},{"conc":320,"http_status_counts":{"200":320},"stream_errors_no_response":0},{"conc":288,"http_status_counts":{"200":288},"stream_errors_no_response":0},{"conc":272,"http_status_counts":{"200":272},"stream_errors_no_response":0},{"conc":280,"http_status_counts":{"200":280},"stream_errors_no_response":0},{"conc":276,"http_status_counts":{"200":276},"stream_errors_no_response":0},{"conc":274,"http_status_counts":{"200":274},"stream_errors_no_response":0},{"conc":273,"http_status_counts":{"200":273},"stream_errors_no_response":0},{"conc":272,"http_status_counts":{"200":272},"stream_errors_no_response":0},{"conc":272,"http_status_counts":{"200":272},"stream_errors_no_response":0},{"conc":264,"http_status_counts":{"200":264},"stream_errors_no_response":0},{"conc":264,"http_status_counts":{"200":264},"stream_errors_no_response":0},{"conc":264,"http_status_counts":{"200":264},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:openai-responses:anthropic","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:gemini:anthropic","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:bedrock:anthropic","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:openai:cohere","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":207,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":423,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":454,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":31,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":216.5942165942166,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":4616.928446771379,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9521305223510478,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":105820,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":17634,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.0010584010584010583,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":408.06640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":620.25390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":620.66015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":620.78125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":622.12109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":66,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.4091809994973029,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.2558773134534622,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":12570,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":808,"first_disqualified_conc":8,"lower_bound":false,"samples":75430},{"bound_ms":5,"rps":{"value":18172,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":3644,"first_disqualified_conc":32,"lower_bound":false,"samples":109045},{"bound_ms":10,"rps":{"value":18962,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":6602,"first_disqualified_conc":64,"lower_bound":false,"samples":113777},{"bound_ms":50,"rps":{"value":20178,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":16502,"first_disqualified_conc":512,"lower_bound":false,"samples":121088},{"bound_ms":100,"rps":{"value":20178,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":16502,"first_disqualified_conc":1024,"lower_bound":false,"samples":121088},{"bound_ms":null,"rps":{"value":20178,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":16502,"first_disqualified_conc":null,"lower_bound":false,"samples":121088}],"sample_counts":{"gateway_c1_samples":24758,"direct_c1_samples":238237},"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:cohere:cohere","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:openai-responses:cohere","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:gemini:cohere","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:bedrock:cohere","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:anthropic:cohere","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:cohere:openai-responses","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:openai:openai-responses","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:anthropic:openai-responses","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:gemini:openai-responses","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:bedrock:openai-responses","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:openai-responses:openai-responses","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:openai:gemini","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":214,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":438,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":468,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":214.77521174373956,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":4656.030795551754,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9710935159643339,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":108858,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":18140,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.001028863289790369,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":408.60546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":622.765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":623.0234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":623.09765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":624.20703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":66,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.26100942658949955,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.38746902504991537,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":18455,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":887,"first_disqualified_conc":16,"lower_bound":false,"samples":110748},{"bound_ms":5,"rps":{"value":18455,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":887,"first_disqualified_conc":32,"lower_bound":false,"samples":110748},{"bound_ms":10,"rps":{"value":19059,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":6510,"first_disqualified_conc":64,"lower_bound":false,"samples":114368},{"bound_ms":50,"rps":{"value":19713,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":18089,"first_disqualified_conc":256,"lower_bound":false,"samples":118296},{"bound_ms":100,"rps":{"value":19713,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":18089,"first_disqualified_conc":1024,"lower_bound":false,"samples":118296},{"bound_ms":null,"rps":{"value":19713,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":18089,"first_disqualified_conc":null,"lower_bound":false,"samples":118296}],"sample_counts":{"gateway_c1_samples":23733,"direct_c1_samples":242637},"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:openai-responses:gemini","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:cohere:gemini","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:anthropic:gemini","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:gemini:gemini","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":193,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":402,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":432,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":195.4768149651111,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5115.696202531646,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9844459642780004,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":121242,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":20204,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.0007588129526071824,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":405.765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":619.2109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":619.58203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":619.78125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":621.0234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":66,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.3557036171891926,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.4672710425108666,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":13349,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":740,"first_disqualified_conc":8,"lower_bound":false,"samples":80104},{"bound_ms":5,"rps":{"value":20568,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":4102,"first_disqualified_conc":64,"lower_bound":false,"samples":123412},{"bound_ms":10,"rps":{"value":21659,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":7445,"first_disqualified_conc":128,"lower_bound":false,"samples":129964},{"bound_ms":50,"rps":{"value":21659,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":7445,"first_disqualified_conc":512,"lower_bound":false,"samples":129964},{"bound_ms":100,"rps":{"value":21659,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":7445,"first_disqualified_conc":1024,"lower_bound":false,"samples":129964},{"bound_ms":null,"rps":{"value":21659,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":7445,"first_disqualified_conc":null,"lower_bound":false,"samples":129964}],"sample_counts":{"gateway_c1_samples":26272,"direct_c1_samples":244214},"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:bedrock:gemini","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:anthropic:bedrock","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:openai-responses:bedrock","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:cohere:bedrock","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:openai:bedrock","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":1038,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":2133,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":2164,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":31,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1041.3099784889591,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":960.3288364249579,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9846268117121296,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":22779,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":3796,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.003512006672812678,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":407.8359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":3411.81640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":3448.46484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":3466.08984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":3477.01171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":10.902433860925841,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.3405578567344605,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":3961,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":3649,"first_disqualified_conc":16,"lower_bound":false,"samples":23772},{"bound_ms":10,"rps":{"value":3961,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":3649,"first_disqualified_conc":32,"lower_bound":false,"samples":23772},{"bound_ms":50,"rps":{"value":3961,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":3649,"first_disqualified_conc":128,"lower_bound":false,"samples":23772},{"bound_ms":100,"rps":{"value":3961,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":3649,"first_disqualified_conc":256,"lower_bound":false,"samples":23772},{"bound_ms":null,"rps":{"value":3961,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":3649,"first_disqualified_conc":null,"lower_bound":false,"samples":23772}],"sample_counts":{"gateway_c1_samples":5167,"direct_c1_samples":237310},"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:gemini:bedrock","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","detail":"Pinned Kong 3.9.4 ai-proxy schema supports OpenAI ingress plus native Gemini and Bedrock llm_format, with native formats restricted to their matching provider. OpenAI ingress is configured for five providers including Cohere v1. Seven exact pairs passed local response and upstream-recorder preflight. OpenAI Responses and native Anthropic/Cohere ingress are not exposed by this ai-proxy schema. Evidence: results/diagnostics/refresh-configuration/kong-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb:bedrock:bedrock","gateway_id":"kong","gateway_build":"kong:3.9.4@sha256:2d572848be6fc58b73ba75edc84f028837bb7be998b3a8b8117fdb44a6892220","run_id":"run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","measured_at":"2026-11-10T17:39:46Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-4.616472104699633,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":91758,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":957,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1988,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":2018,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1094.7058552964445,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":913.4873949579832,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9885933476042671,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":21741,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":3623,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.006669426429327078,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":407.671875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":3782.19140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":3821.51953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":3840.78515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":3862.0625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":46.30672272130818,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.3255040970998626,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":3922,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4731,"first_disqualified_conc":16,"lower_bound":false,"samples":23539},{"bound_ms":10,"rps":{"value":3922,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":4731,"first_disqualified_conc":16,"lower_bound":false,"samples":23539},{"bound_ms":50,"rps":{"value":4006,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":38529,"first_disqualified_conc":128,"lower_bound":false,"samples":24042},{"bound_ms":100,"rps":{"value":4006,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":38529,"first_disqualified_conc":256,"lower_bound":false,"samples":24042},{"bound_ms":null,"rps":{"value":4006,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":38529,"first_disqualified_conc":32768,"lower_bound":false,"samples":24042}],"sample_counts":{"gateway_c1_samples":5584,"direct_c1_samples":241921},"provenance":{"source_url":"https://onthebench.ai/gateways/kong","raw_result_url":"https://onthebench.ai/api/v1/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-kong-d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"d4b6d413ea59fd281dacf70be5988600ace51a15e8665c6c7dedcc87dd2cd2fb"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:cohere:cohere","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":11850,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":12623,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":12651,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":12063.492063492064,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":82.89473684210526,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.24961199687176264,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":504,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":83,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":6,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.051587301587301584,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":592.81640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":663.33203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":663.33203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":663.33203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":663.33203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":81,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":83,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":12764,"first_disqualified_conc":4,"lower_bound":false,"samples":499},{"bound_ms":100,"rps":{"value":83,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":12764,"first_disqualified_conc":8,"lower_bound":false,"samples":499},{"bound_ms":null,"rps":{"value":112,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1024,"p99_us":3940819,"first_disqualified_conc":8192,"lower_bound":false,"samples":674}],"sample_counts":{"gateway_c1_samples":503,"direct_c1_samples":264219},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:openai:cohere","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":301,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":7551,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":8314,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":8341,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":7593.984962406015,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":131.68316831683168,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2502569364427439,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":798,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":132,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.03132832080200501,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":592.56640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":666.06640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":666.0703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":666.0703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":666.0703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":61,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.0020496859180001844,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":131,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":8409,"first_disqualified_conc":2,"lower_bound":false,"samples":787},{"bound_ms":50,"rps":{"value":131,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":8409,"first_disqualified_conc":8,"lower_bound":false,"samples":787},{"bound_ms":100,"rps":{"value":131,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":8409,"first_disqualified_conc":16,"lower_bound":false,"samples":787},{"bound_ms":null,"rps":{"value":138,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":510607,"first_disqualified_conc":16384,"lower_bound":false,"samples":829}],"sample_counts":{"gateway_c1_samples":790,"direct_c1_samples":267951},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:bedrock:cohere","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","detail":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:anthropic:cohere","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":8248,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":9054,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":9082,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":8407.202216066482,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":118.94563426688632,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2504942972604469,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":722,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":120,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.024930747922437674,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":592.1796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":665.765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":665.765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":665.765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":665.765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":61,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":118,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":9145,"first_disqualified_conc":2,"lower_bound":false,"samples":713},{"bound_ms":50,"rps":{"value":120,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":17574,"first_disqualified_conc":8,"lower_bound":false,"samples":722},{"bound_ms":100,"rps":{"value":120,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":17574,"first_disqualified_conc":16,"lower_bound":false,"samples":722},{"bound_ms":null,"rps":{"value":120,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":17574,"first_disqualified_conc":16384,"lower_bound":false,"samples":722}],"sample_counts":{"gateway_c1_samples":723,"direct_c1_samples":260556},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:openai-responses:cohere","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":304,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":9710,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":10423,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":10451,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":9886.363636363636,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":101.14942528735632,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.25085857203068684,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":616,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":102,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.04707792207792208,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":592.70703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":667.8046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":667.8046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":667.8046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":667.8046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":62,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":304,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":104,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":20420,"first_disqualified_conc":8,"lower_bound":false,"samples":630},{"bound_ms":100,"rps":{"value":104,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":20420,"first_disqualified_conc":16,"lower_bound":false,"samples":630},{"bound_ms":null,"rps":{"value":106,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":647637,"first_disqualified_conc":8192,"lower_bound":false,"samples":637}],"sample_counts":{"gateway_c1_samples":615,"direct_c1_samples":263786},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:gemini:cohere","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":13881,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":14770,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":14798,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":14342.723004694835,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":69.72176759410802,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2507360988510164,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":426,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":70,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.06338028169014084,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":593.359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":670.828125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":670.9921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":670.9921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":670.9921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":63,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.09640675558624215,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":71,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":14828,"first_disqualified_conc":4,"lower_bound":false,"samples":427},{"bound_ms":100,"rps":{"value":71,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":14828,"first_disqualified_conc":8,"lower_bound":false,"samples":427},{"bound_ms":null,"rps":{"value":73,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":3307564,"first_disqualified_conc":8192,"lower_bound":false,"samples":439}],"sample_counts":{"gateway_c1_samples":428,"direct_c1_samples":262194},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:cohere:anthropic","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","detail":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:bedrock:anthropic","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","detail":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:openai-responses:anthropic","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":13061,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":13826,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":13854,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":13414.096916299559,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":74.54844006568145,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.25025039298228663,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":454,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":75,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.05506607929515418,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":592.65625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":667.640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":667.640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":667.640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":667.640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":62,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":76,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":13878,"first_disqualified_conc":4,"lower_bound":false,"samples":458},{"bound_ms":100,"rps":{"value":76,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":13878,"first_disqualified_conc":8,"lower_bound":false,"samples":458},{"bound_ms":null,"rps":{"value":76,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":13878,"first_disqualified_conc":8192,"lower_bound":false,"samples":458}],"sample_counts":{"gateway_c1_samples":457,"direct_c1_samples":263938},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:gemini:anthropic","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":305,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":16324,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":17226,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":17254,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":16831.955922865014,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":59.41080196399345,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2501531621486448,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":363,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":60,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.0743801652892562,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":591.91796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":671.87109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":671.87109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":671.87109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":671.87109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":63,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":305,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":60,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":33932,"first_disqualified_conc":4,"lower_bound":false,"samples":362},{"bound_ms":100,"rps":{"value":60,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":33932,"first_disqualified_conc":8,"lower_bound":false,"samples":362},{"bound_ms":null,"rps":{"value":60,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":33932,"first_disqualified_conc":4096,"lower_bound":false,"samples":362}],"sample_counts":{"gateway_c1_samples":364,"direct_c1_samples":262033},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:openai:anthropic","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":304,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":10313,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":11125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":11153,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":10411.663807890223,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":96.04612850082371,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2500487063572403,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":583,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":97,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.03773584905660377,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":13428,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":14151,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":3,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":1348,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":1232.8623220270053,"certified":true,"suppressed":false,"headroom":0.20745279457185184,"rig_ceiling":5942.857142857143,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":117,"certified":true,"suppressed":false,"headroom":0.20745279457185184,"rig_ceiling":5942.857142857143,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":592.48828125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":667.12109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":667.12109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":667.12109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":667.12109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":62,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":304,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":97,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":10954,"first_disqualified_conc":8,"lower_bound":false,"samples":586},{"bound_ms":100,"rps":{"value":98,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":88352,"first_disqualified_conc":16,"lower_bound":false,"samples":589},{"bound_ms":null,"rps":{"value":98,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":88352,"first_disqualified_conc":16384,"lower_bound":false,"samples":589}],"sample_counts":{"gateway_c1_samples":579,"direct_c1_samples":265135,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":96,"http_status_counts":{"200":96},"stream_errors_no_response":0},{"conc":112,"http_status_counts":{"200":112},"stream_errors_no_response":0},{"conc":120,"http_status_counts":{"200":120},"stream_errors_no_response":0},{"conc":116,"http_status_counts":{"200":116},"stream_errors_no_response":0},{"conc":118,"http_status_counts":{"200":118},"stream_errors_no_response":0},{"conc":117,"http_status_counts":{"200":117},"stream_errors_no_response":0},{"conc":117,"http_status_counts":{"200":117},"stream_errors_no_response":0},{"conc":117,"http_status_counts":{"200":117},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:anthropic:anthropic","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":10994,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":11745,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":11773,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":11322.160148975792,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":88.32236842105263,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2501294799366864,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":537,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":89,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":268,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.03910614525139665,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":14510,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":15815,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21123us) came in under the mock's (21136us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":3187.580899184528,"certified":true,"suppressed":false,"headroom":0.2460999958929231,"rig_ceiling":12952.380952380952,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":255,"certified":true,"suppressed":false,"headroom":0.2460999958929231,"rig_ceiling":12952.380952380952,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":592.48828125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":658.05859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":658.05859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":658.05859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":658.05859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":62,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":89,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":23288,"first_disqualified_conc":8,"lower_bound":false,"samples":538},{"bound_ms":100,"rps":{"value":89,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":23288,"first_disqualified_conc":16,"lower_bound":false,"samples":538},{"bound_ms":null,"rps":{"value":91,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":796228,"first_disqualified_conc":4096,"lower_bound":false,"samples":550}],"sample_counts":{"gateway_c1_samples":544,"direct_c1_samples":263642,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":192,"http_status_counts":{"200":192},"stream_errors_no_response":0},{"conc":224,"http_status_counts":{"200":224},"stream_errors_no_response":0},{"conc":240,"http_status_counts":{"200":240},"stream_errors_no_response":0},{"conc":248,"http_status_counts":{"200":248},"stream_errors_no_response":0},{"conc":252,"http_status_counts":{"200":252},"stream_errors_no_response":0},{"conc":254,"http_status_counts":{"200":254},"stream_errors_no_response":0},{"conc":255,"http_status_counts":{"200":255},"stream_errors_no_response":0},{"conc":255,"http_status_counts":{"200":255},"stream_errors_no_response":0},{"conc":255,"http_status_counts":{"200":255},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:openai-responses:bedrock","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":301,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":17646,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":18361,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":18388,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":18244.04761904762,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":54.81239804241436,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2502884887591485,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":336,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":55,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":17,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.0744047619047619,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":592.97265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":689.4375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":689.4375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":689.4375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":689.4375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":64,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":56,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":18407,"first_disqualified_conc":4,"lower_bound":false,"samples":338},{"bound_ms":100,"rps":{"value":56,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":18407,"first_disqualified_conc":8,"lower_bound":false,"samples":338},{"bound_ms":null,"rps":{"value":57,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2048,"p99_us":5344748,"first_disqualified_conc":4096,"lower_bound":false,"samples":345}],"sample_counts":{"gateway_c1_samples":339,"direct_c1_samples":267586},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:cohere:bedrock","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","detail":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:anthropic:bedrock","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":305,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":16315,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":17156,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":17184,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":18047.337278106508,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":55.40983606557377,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.25012322825186667,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":338,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":56,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":17,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.07396449704142012,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":592.72265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":687.4609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":687.4609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":687.4609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":687.4609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":63,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":305,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":62,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":34711,"first_disqualified_conc":4,"lower_bound":false,"samples":376},{"bound_ms":100,"rps":{"value":62,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":34711,"first_disqualified_conc":8,"lower_bound":false,"samples":376},{"bound_ms":null,"rps":{"value":63,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":64,"p99_us":1110053,"first_disqualified_conc":4096,"lower_bound":false,"samples":382}],"sample_counts":{"gateway_c1_samples":366,"direct_c1_samples":262394},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:gemini:bedrock","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":21239,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":22150,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":22177,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":22086.330935251797,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":45.276872964169385,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2506164650746462,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":278,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":46,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":17,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.07913669064748201,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":591.921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":692.7734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":692.77734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":692.77734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":692.77734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":64,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.005149277000641089,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":46,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":22251,"first_disqualified_conc":4,"lower_bound":false,"samples":280},{"bound_ms":100,"rps":{"value":46,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":22251,"first_disqualified_conc":8,"lower_bound":false,"samples":280},{"bound_ms":null,"rps":{"value":46,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":22251,"first_disqualified_conc":4096,"lower_bound":false,"samples":280}],"sample_counts":{"gateway_c1_samples":281,"direct_c1_samples":267100},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:bedrock:bedrock","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":303,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":10032,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":10847,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":10874,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":10134.003350083753,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":98.67768595041322,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.24972426540447598,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":597,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":99,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.04522613065326633,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":592.5078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":671.40234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":671.40234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":671.40234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":671.40234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":61,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":303,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":101,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":20685,"first_disqualified_conc":8,"lower_bound":false,"samples":607},{"bound_ms":100,"rps":{"value":101,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":20685,"first_disqualified_conc":16,"lower_bound":false,"samples":607},{"bound_ms":null,"rps":{"value":104,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":1374776,"first_disqualified_conc":16384,"lower_bound":false,"samples":625}],"sample_counts":{"gateway_c1_samples":595,"direct_c1_samples":267885},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:openai:bedrock","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":14005,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":14745,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":14772,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":14262.295081967213,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":70.11494252873564,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.24985541302136363,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":427,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":71,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":17,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.0468384074941452,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":593.1875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":689.1015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":689.1015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":689.1015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":689.1015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":63,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":72,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":28461,"first_disqualified_conc":4,"lower_bound":false,"samples":434},{"bound_ms":100,"rps":{"value":72,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":28461,"first_disqualified_conc":8,"lower_bound":false,"samples":434},{"bound_ms":null,"rps":{"value":72,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":28461,"first_disqualified_conc":8192,"lower_bound":false,"samples":434}],"sample_counts":{"gateway_c1_samples":427,"direct_c1_samples":272464},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:openai:openai","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":8512,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":9522,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":9549,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":8644.793152639088,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":115.67656765676568,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.24990558455671374,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":701,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":116,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":32,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.028530670470756064,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":11154,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":11974,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21070us) came in under the mock's (21072us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":9,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":1207.9948436569616,"certified":true,"suppressed":false,"headroom":0.3835870723305876,"rig_ceiling":3149.2063492063494,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":62,"certified":true,"suppressed":false,"headroom":0.3835870723305876,"rig_ceiling":3149.2063492063494,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":592.95703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":628.60546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":628.60546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":628.60546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":628.60546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":61,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":117,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":9350,"first_disqualified_conc":2,"lower_bound":false,"samples":708},{"bound_ms":50,"rps":{"value":119,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":17670,"first_disqualified_conc":8,"lower_bound":false,"samples":718},{"bound_ms":100,"rps":{"value":119,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":17670,"first_disqualified_conc":16,"lower_bound":false,"samples":718},{"bound_ms":null,"rps":{"value":119,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":17670,"first_disqualified_conc":16384,"lower_bound":false,"samples":718}],"sample_counts":{"gateway_c1_samples":701,"direct_c1_samples":267297,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":48,"http_status_counts":{"200":48},"stream_errors_no_response":0},{"conc":56,"http_status_counts":{"200":56},"stream_errors_no_response":0},{"conc":60,"http_status_counts":{"200":60},"stream_errors_no_response":0},{"conc":62,"http_status_counts":{"200":62},"stream_errors_no_response":0},{"conc":63,"http_status_counts":{"200":63},"stream_errors_no_response":0},{"conc":62,"http_status_counts":{"200":62},"stream_errors_no_response":0},{"conc":62,"http_status_counts":{"200":62},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:anthropic:openai","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"not_served","raw_state":"failed","reason":"failed","detail":"probed and answered HTTP 200, but the body is not a usable anthropic response (see body_snippet)","evidence_scope":"Recorded run evidence","probe_http_status":"200","response_excerpt":"the 2xx body is not a response of the anthropic dialect with assistant text at content[type=text].text: {\"id\":\"resp_bGl0ZWxsbTpjdXN0b21fbGxtX3Byb3ZpZGVyOm9wZW5haTttb2RlbF9pZDo2MTk3ZDQ1OWNlMzc1OTE2NzVlMTIyMTg3ZDAwOTViNzIyNDhmMTRmYTUwNzQzMDZkMzFlMzlhODkzMTM2OWQ4O3Jlc3BvbnNlX2lkOmNoYXRjbXBsLXg=\",\"type\":\"message\",\"role\":\"assistant\",\"model\":\"gpt-4o-mini\",\"stop_sequence\":null,\"usage\":{\"input_tokens\":10,\"output_tokens\":2},\"content\":[],\"stop_reason\":\"end_turn\",\"stop_details\":null}"},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:openai-responses:openai","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":304,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":10676,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":11606,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":11634,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":10555.555555555555,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":94.73684210526315,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2503730077247171,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":576,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":95,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.059027777777777776,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":591.90625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":629.95703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":629.96875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":629.96875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":629.96875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":62,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.00555168581824748,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":304,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":95,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":22974,"first_disqualified_conc":8,"lower_bound":false,"samples":572},{"bound_ms":100,"rps":{"value":95,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":22974,"first_disqualified_conc":16,"lower_bound":false,"samples":572},{"bound_ms":null,"rps":{"value":95,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":22974,"first_disqualified_conc":4096,"lower_bound":false,"samples":572}],"sample_counts":{"gateway_c1_samples":559,"direct_c1_samples":265389},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:cohere:openai","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","detail":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:bedrock:openai","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","detail":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:gemini:openai","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":14407,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":15293,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":15321,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":15086.41975308642,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":66.28477905073649,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.25038840963155784,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":405,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":67,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.044444444444444446,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":592.00390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":632.30859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":632.31640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":632.31640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":632.31640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":63,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.003008508378215942,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":68,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":15465,"first_disqualified_conc":4,"lower_bound":false,"samples":410},{"bound_ms":100,"rps":{"value":68,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":15465,"first_disqualified_conc":8,"lower_bound":false,"samples":410},{"bound_ms":null,"rps":{"value":68,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":15465,"first_disqualified_conc":8192,"lower_bound":false,"samples":410}],"sample_counts":{"gateway_c1_samples":412,"direct_c1_samples":263451},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:cohere:openai-responses","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","detail":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:bedrock:openai-responses","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","detail":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:gemini:openai-responses","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":306,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":17577,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":18499,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":18527,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":18005.865102639294,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":55.537459283387626,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.250691676136018,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":341,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":56,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.08211143695014662,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":592.40625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":674.6484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":674.6484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":674.6484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":674.6484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":63,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":306,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":56,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":18534,"first_disqualified_conc":4,"lower_bound":false,"samples":337},{"bound_ms":100,"rps":{"value":56,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":18534,"first_disqualified_conc":8,"lower_bound":false,"samples":337},{"bound_ms":null,"rps":{"value":59,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2048,"p99_us":5599008,"first_disqualified_conc":4096,"lower_bound":false,"samples":360}],"sample_counts":{"gateway_c1_samples":338,"direct_c1_samples":262004},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:anthropic:openai-responses","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":303,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":13287,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":14010,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":14038,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":13738.738738738739,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":72.78688524590164,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.25008979839096346,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":444,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":73,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.07207207207207207,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":591.65234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":661.2265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":661.2265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":661.2265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":661.2265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":63,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":303,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":74,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":13937,"first_disqualified_conc":4,"lower_bound":false,"samples":450},{"bound_ms":100,"rps":{"value":74,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":13937,"first_disqualified_conc":8,"lower_bound":false,"samples":450},{"bound_ms":null,"rps":{"value":74,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":13937,"first_disqualified_conc":8192,"lower_bound":false,"samples":450}],"sample_counts":{"gateway_c1_samples":449,"direct_c1_samples":260854},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:openai:openai-responses","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":305,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":10726,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":11474,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":11502,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":10799.289520426288,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":92.59868421052632,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2502330796843856,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":563,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":93,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.05328596802841918,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":591.7890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":669.40234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":669.40234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":669.40234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":669.40234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":62,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":305,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":92,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":11558,"first_disqualified_conc":8,"lower_bound":false,"samples":556},{"bound_ms":100,"rps":{"value":92,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":11558,"first_disqualified_conc":16,"lower_bound":false,"samples":556},{"bound_ms":null,"rps":{"value":93,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":183258,"first_disqualified_conc":16384,"lower_bound":false,"samples":559}],"sample_counts":{"gateway_c1_samples":557,"direct_c1_samples":261925},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:openai-responses:openai-responses","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":11554,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":12280,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":12308,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":11692.307692307691,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":85.52631578947368,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2497786700112885,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":520,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":86,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.06346153846153846,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":592.46484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":627.4453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":629.47265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":629.47265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":629.47265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":62,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":3.0355931092621313,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":87,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":23671,"first_disqualified_conc":8,"lower_bound":false,"samples":524},{"bound_ms":100,"rps":{"value":87,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":23671,"first_disqualified_conc":16,"lower_bound":false,"samples":524},{"bound_ms":null,"rps":{"value":88,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":32,"p99_us":556619,"first_disqualified_conc":8192,"lower_bound":false,"samples":529}],"sample_counts":{"gateway_c1_samples":518,"direct_c1_samples":265921},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:openai-responses:gemini","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":304,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":10486,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":11201,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":11229,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":10628.272251308901,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":94.08866995073892,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.25043632937727195,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":573,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":95,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.041884816753926704,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":592.390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":669.41015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":669.4140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":669.4140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":669.4140625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":62,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":304,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":96,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":21588,"first_disqualified_conc":8,"lower_bound":false,"samples":578},{"bound_ms":100,"rps":{"value":96,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":21588,"first_disqualified_conc":16,"lower_bound":false,"samples":578},{"bound_ms":null,"rps":{"value":96,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":21588,"first_disqualified_conc":16384,"lower_bound":false,"samples":578}],"sample_counts":{"gateway_c1_samples":570,"direct_c1_samples":265239},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:gemini:gemini","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":10073,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":10889,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":10916,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":10236.486486486487,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":97.6897689768977,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2501839420461313,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":592,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":98,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.048986486486486486,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":591.546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":627.6875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":627.6875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":627.6875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":627.6875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":61,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":101,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":21419,"first_disqualified_conc":8,"lower_bound":false,"samples":612},{"bound_ms":100,"rps":{"value":101,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":21419,"first_disqualified_conc":16,"lower_bound":false,"samples":612},{"bound_ms":null,"rps":{"value":101,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":21419,"first_disqualified_conc":4096,"lower_bound":false,"samples":612}],"sample_counts":{"gateway_c1_samples":592,"direct_c1_samples":267671},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:cohere:gemini","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","detail":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:bedrock:gemini","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","detail":"Pinned v1.108.3 source b7d579fab2818968322ef96e5b8e98dc8178fafc: Chat, Responses, Messages and Gemini ingress translate to all six configured upstream wires. The per-model upstream setting selects Chat versus Responses without changing workload parameters. The default Messages-to-Chat bridge reaches the correct wire but returns empty content; it is configured as shipped, without a global bridge override, so its failure is recorded. Native Cohere and Bedrock ingress use the shipped /cohere and /bedrock provider-prefixed passthrough routes in litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py; passthrough is same-dialect only. Configured pairs that fail are recorded as failures. Evidence: results/diagnostics/refresh-configuration/litellm-python-final-config-correctness-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:openai:gemini","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":8225,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":8995,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":9023,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":8281.036834924966,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":120.75782537067545,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.25021912323444456,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":733,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":122,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.0286493860845839,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":591.8359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":665.40625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":665.40625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":665.40625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":665.40625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":61,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":121,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":8923,"first_disqualified_conc":2,"lower_bound":false,"samples":732},{"bound_ms":50,"rps":{"value":123,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":17053,"first_disqualified_conc":8,"lower_bound":false,"samples":740},{"bound_ms":100,"rps":{"value":123,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":17053,"first_disqualified_conc":16,"lower_bound":false,"samples":740},{"bound_ms":null,"rps":{"value":127,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":128,"p99_us":1100645,"first_disqualified_conc":8192,"lower_bound":false,"samples":767}],"sample_counts":{"gateway_c1_samples":725,"direct_c1_samples":265737},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf:anthropic:gemini","gateway_id":"litellm-python","gateway_build":"ghcr.io/berriai/litellm:v1.108.3@sha256:e2bce769eb29b100d53ce90cbf2d6bba00dce82cb894c79d092ef6377523d7e5","run_id":"run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","measured_at":"2026-11-10T17:39:41Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":2.8326697782721233,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":98924,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":8964,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":9686,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":9714,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":9144.144144144144,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":109.35960591133005,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.25073457424131973,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":666,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":110,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":13,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.03453453453453453,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":593.02734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":666.55078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":666.5546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":666.5546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":666.5546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":61,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.0057259229747044565,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":111,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":9727,"first_disqualified_conc":2,"lower_bound":false,"samples":667},{"bound_ms":50,"rps":{"value":113,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":19230,"first_disqualified_conc":8,"lower_bound":false,"samples":682},{"bound_ms":100,"rps":{"value":113,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":19230,"first_disqualified_conc":16,"lower_bound":false,"samples":682},{"bound_ms":null,"rps":{"value":113,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":19230,"first_disqualified_conc":16384,"lower_bound":false,"samples":682}],"sample_counts":{"gateway_c1_samples":666,"direct_c1_samples":263751},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-python","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-python-10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"10bb57507b26d310b59f12956034785b615cd5338cdaf5cee0e587c350c53acf"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:anthropic:bedrock","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:cohere:bedrock","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:bedrock:bedrock","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:gemini:bedrock","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:openai:bedrock","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_served","raw_state":"failed","reason":"failed","detail":"probed and answered failed (HTTP 501)","evidence_scope":"Recorded run evidence","probe_http_status":"501","response_excerpt":"{\"error\":{\"message\":\"unsupported by the rust path: unrecognized request parameter\",\"type\":\"api_error\",\"param\":null,\"code\":501}}"},"test":{"ingress_protocol":"openai","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:openai-responses:bedrock","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:bedrock:openai","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:openai:openai","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{"added_latency_p50_us":{"value":249,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":279,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":306,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":272.6013913681787,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":3668.359853121175,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.6790579018170796,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":59941,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":9989,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"no stream frame arrived from the gateway: it answered HTTP 501, not 2xx, so what it sent is an error response rather than a stream to time","state":"not_measured","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"no stream frame arrived from the gateway: it answered HTTP 501, not 2xx, so what it sent is an error response rather than a stream to time","state":"not_measured","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"no stream frame arrived from the gateway: it answered HTTP 501, not 2xx, so what it sent is an error response rather than a stream to time","state":"not_measured","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"no stream frame arrived from the gateway: it answered HTTP 501, not 2xx, so what it sent is an error response rather than a stream to time","state":"not_measured","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":0,"certified":true,"suppressed":false,"note":"measured_failure","state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":0,"certified":true,"suppressed":false,"note":"measured_failure","state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":12.73828125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":18.9921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":18.99609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":19.875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":19.484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.1791634760562065,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":8404,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":588,"first_disqualified_conc":8,"lower_bound":false,"samples":50435},{"bound_ms":5,"rps":{"value":10049,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2486,"first_disqualified_conc":32,"lower_bound":false,"samples":60306},{"bound_ms":10,"rps":{"value":10049,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2486,"first_disqualified_conc":64,"lower_bound":false,"samples":60306},{"bound_ms":50,"rps":{"value":10049,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2486,"first_disqualified_conc":256,"lower_bound":false,"samples":60306},{"bound_ms":100,"rps":{"value":10049,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2486,"first_disqualified_conc":1024,"lower_bound":false,"samples":60306},{"bound_ms":null,"rps":{"value":10049,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2486,"first_disqualified_conc":null,"lower_bound":false,"samples":60306}],"sample_counts":{"gateway_c1_samples":22021,"direct_c1_samples":267021},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"501":1},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:anthropic:openai","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:openai-responses:openai","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:gemini:openai","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:cohere:openai","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:gemini:openai-responses","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:cohere:openai-responses","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:bedrock:openai-responses","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:openai-responses:openai-responses","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{"added_latency_p50_us":{"value":267,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":298,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":325,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":287.74399582885263,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":3475.3114382785957,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7339219155366118,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":61374,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":10227,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":12.7421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":19.47265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":19.28515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":19.5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":19.5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":66,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.028056430633918075,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":8050,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":705,"first_disqualified_conc":8,"lower_bound":false,"samples":48309},{"bound_ms":5,"rps":{"value":10191,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2455,"first_disqualified_conc":32,"lower_bound":false,"samples":61157},{"bound_ms":10,"rps":{"value":10191,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2455,"first_disqualified_conc":64,"lower_bound":false,"samples":61157},{"bound_ms":50,"rps":{"value":10191,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2455,"first_disqualified_conc":256,"lower_bound":false,"samples":61157},{"bound_ms":100,"rps":{"value":10191,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2455,"first_disqualified_conc":1024,"lower_bound":false,"samples":61157},{"bound_ms":null,"rps":{"value":10191,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2455,"first_disqualified_conc":null,"lower_bound":false,"samples":61157}],"sample_counts":{"gateway_c1_samples":20661,"direct_c1_samples":267569},"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:openai:openai-responses","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:anthropic:openai-responses","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:openai:anthropic","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{"added_latency_p50_us":{"value":253,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":283,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":311,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":277.0705901801951,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":3609.1885441527443,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.6965431268306815,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":60490,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":10080,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"no stream frame arrived from the gateway: it answered HTTP 501, not 2xx, so what it sent is an error response rather than a stream to time","state":"not_measured","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"no stream frame arrived from the gateway: it answered HTTP 501, not 2xx, so what it sent is an error response rather than a stream to time","state":"not_measured","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"no stream frame arrived from the gateway: it answered HTTP 501, not 2xx, so what it sent is an error response rather than a stream to time","state":"not_measured","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"no stream frame arrived from the gateway: it answered HTTP 501, not 2xx, so what it sent is an error response rather than a stream to time","state":"not_measured","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":0,"certified":true,"suppressed":false,"note":"measured_failure","state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":0,"certified":true,"suppressed":false,"note":"measured_failure","state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":12.7578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":19.046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":18.984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":19.66015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":19.33203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.3275657622729096,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":8572,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":594,"first_disqualified_conc":8,"lower_bound":false,"samples":51441},{"bound_ms":5,"rps":{"value":10082,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2482,"first_disqualified_conc":32,"lower_bound":false,"samples":60500},{"bound_ms":10,"rps":{"value":10082,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2482,"first_disqualified_conc":64,"lower_bound":false,"samples":60500},{"bound_ms":50,"rps":{"value":10082,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2482,"first_disqualified_conc":256,"lower_bound":false,"samples":60500},{"bound_ms":100,"rps":{"value":10082,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2482,"first_disqualified_conc":1024,"lower_bound":false,"samples":60500},{"bound_ms":null,"rps":{"value":10082,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2482,"first_disqualified_conc":null,"lower_bound":false,"samples":60500}],"sample_counts":{"gateway_c1_samples":21651,"direct_c1_samples":262942},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"501":1},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:cohere:anthropic","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:bedrock:anthropic","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:gemini:anthropic","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:anthropic:anthropic","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{"added_latency_p50_us":{"value":275,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":304,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":332,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":297.0902986640554,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":3365.97998822837,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.7061370243602647,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":57188,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":9530,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":406,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":477,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":2,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":3,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":25279.8938149557,"certified":true,"suppressed":false,"headroom":0.5415646457910123,"rig_ceiling":46679.36507936508,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":919,"certified":true,"suppressed":false,"headroom":0.5415646457910123,"rig_ceiling":46679.36507936508,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":12.7421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":20.3515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":20.21484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":21.0078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":20.5625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.01099763747584857,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":8878,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":606,"first_disqualified_conc":8,"lower_bound":false,"samples":53276},{"bound_ms":5,"rps":{"value":9865,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2555,"first_disqualified_conc":32,"lower_bound":false,"samples":59196},{"bound_ms":10,"rps":{"value":9865,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2555,"first_disqualified_conc":64,"lower_bound":false,"samples":59196},{"bound_ms":50,"rps":{"value":9865,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2555,"first_disqualified_conc":256,"lower_bound":false,"samples":59196},{"bound_ms":100,"rps":{"value":9865,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2555,"first_disqualified_conc":1024,"lower_bound":false,"samples":59196},{"bound_ms":null,"rps":{"value":9865,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2555,"first_disqualified_conc":null,"lower_bound":false,"samples":59196}],"sample_counts":{"gateway_c1_samples":20084,"direct_c1_samples":262977,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":768,"http_status_counts":{"200":768},"stream_errors_no_response":0},{"conc":896,"http_status_counts":{"200":896},"stream_errors_no_response":0},{"conc":960,"http_status_counts":{"200":960},"stream_errors_no_response":0},{"conc":928,"http_status_counts":{"200":928},"stream_errors_no_response":0},{"conc":912,"http_status_counts":{"200":912},"stream_errors_no_response":0},{"conc":920,"http_status_counts":{"200":920},"stream_errors_no_response":0},{"conc":916,"http_status_counts":{"200":916},"stream_errors_no_response":0},{"conc":918,"http_status_counts":{"200":918},"stream_errors_no_response":0},{"conc":919,"http_status_counts":{"200":919},"stream_errors_no_response":0},{"conc":919,"http_status_counts":{"200":919},"stream_errors_no_response":0},{"conc":919,"http_status_counts":{"200":919},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:openai-responses:anthropic","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:anthropic:gemini","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:cohere:gemini","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:gemini:gemini","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:bedrock:gemini","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:openai-responses:gemini","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:openai:gemini","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:openai-responses:cohere","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:openai:cohere","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:cohere:cohere","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:gemini:cohere","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:anthropic:cohere","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4:bedrock:cohere","gateway_id":"litellm-rust","gateway_build":"src/litellm-rust/target/release/litellm-gateway","run_id":"run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","measured_at":"2026-11-10T17:39:51Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","detail":"Pinned v1.108.3 standalone Rust source b7d579fab2818968322ef96e5b8e98dc8178fafc: gateway-inference/src/lib.rs registers Chat, Responses and Messages ingress. inference-chat/src/common_utils.rs registers OpenAI-like, Anthropic and Bedrock Converse adapters; inference-responses/src/prepare.rs supports native OpenAI Responses; inference-messages/src/common_utils.rs has native Anthropic, but no OpenAI cross-conversion. Its Bedrock Messages path uses Invoke, not this matrix's Converse wire. Native Gemini, Cohere and Bedrock ingress are not registered. OpenAI Chat to Bedrock Converse is configured and probed with the standard workload request, which includes max_tokens; the released gateway answers HTTP 501 (\"unsupported by the rust path: unrecognized request parameter\"), which the engine records as not configured. Pairs outside the routes above are not configured. Evidence: results/diagnostics/refresh-configuration/litellm-rust-exploratory-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.363964282372997,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83343,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535,"gateway_provenance":{"binary_name":"litellm-gateway","binary_sha256":"ae1fec119755a8a2ae5b6a9d56225c34205ce7a54548cb11696b03b2557ce01b","cargo_lock_sha256":"ae9550da9f00c3ffb10ac2974d7d33c3784902c702c74c6aa85f1b83f1efcb2d","compiler":"rustc 1.99.1 (58cf8f9c3 2026-10-13)\nbinary: rustc\ncommit-hash: 58cf8f9c35b355199727627d07ae34a4e2846d67\ncommit-date: 2026-10-13\nhost: aarch64-unknown-linux-gnu\nrelease: 1.99.1\nLLVM version: 23.1.1","source_commit":"b7d579fab2818968322ef96e5b8e98dc8178fafc","source_dirty":false}},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/litellm-rust","raw_result_url":"https://onthebench.ai/api/v1/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-litellm-rust-69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"69d8e6ed325740d9da7f0aaf31b815418c4801eb6549b2e2c68f768a92af3ad4"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:anthropic:openai-responses","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:openai:openai-responses","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:gemini:openai-responses","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:cohere:openai-responses","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:openai-responses:openai-responses","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:bedrock:openai-responses","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:openai-responses:gemini","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:cohere:gemini","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:bedrock:gemini","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:anthropic:gemini","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:gemini:gemini","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:openai:gemini","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":34189,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":34903,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":34941,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":38,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":3395.3488372093025,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":294.52054794520546,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.02833912022723337,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":215,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":35,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":84,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":80.31640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":144.08203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":141.51171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":144.8203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":144.71875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":120,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.4419008197056353,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.1152909126823189,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":42200,"first_disqualified_conc":2,"lower_bound":false,"samples":181},{"bound_ms":100,"rps":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":52183,"first_disqualified_conc":2,"lower_bound":false,"samples":181},{"bound_ms":null,"rps":{"value":38,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2474410,"first_disqualified_conc":32,"lower_bound":false,"samples":230}],"sample_counts":{"gateway_c1_samples":178,"direct_c1_samples":213320},"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:bedrock:cohere","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:gemini:cohere","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:anthropic:cohere","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:cohere:cohere","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:openai-responses:cohere","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:openai:cohere","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":307,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":34053,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":42953,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":42981,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":3063.063063063063,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":326.4705882352941,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.02686924016940405,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":222,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":36,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":79,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":86.46875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":141.58984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":134.26953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":142.06640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":142.01953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":127,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.8110074898737798,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":307,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.11088631518992631,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":29,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":34152,"first_disqualified_conc":2,"lower_bound":false,"samples":178},{"bound_ms":100,"rps":{"value":29,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":34152,"first_disqualified_conc":2,"lower_bound":false,"samples":178},{"bound_ms":null,"rps":{"value":36,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":1430027,"first_disqualified_conc":16,"lower_bound":false,"samples":220}],"sample_counts":{"gateway_c1_samples":178,"direct_c1_samples":265039},"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:openai:bedrock","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"bedrock","upstream_transport":"tls","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":303,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":35450,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":35742,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":35771,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":29,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":4976.076555023923,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":200.96153846153845,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.040498520437646256,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":209,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":34,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":89,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":85.7578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":348.8046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":332.67578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":349.359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":349.30859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":174,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.6047438141000504,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":303,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.11095193717665752,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":43372,"first_disqualified_conc":2,"lower_bound":false,"samples":172},{"bound_ms":100,"rps":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":43372,"first_disqualified_conc":2,"lower_bound":false,"samples":172},{"bound_ms":null,"rps":{"value":34,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":1561239,"first_disqualified_conc":16,"lower_bound":false,"samples":209}],"sample_counts":{"gateway_c1_samples":170,"direct_c1_samples":258581},"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:cohere:bedrock","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:gemini:bedrock","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:anthropic:bedrock","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:bedrock:bedrock","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:openai-responses:bedrock","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:openai:anthropic","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":301,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":34189,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":1029936,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":1029964,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":3348.8372093023254,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":298.61111111111114,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.02727559585540872,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":215,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":35,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":214,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":751,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":1758,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":7,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":4429.711949178542,"certified":true,"suppressed":false,"headroom":0.822735415089175,"rig_ceiling":5384.126984126984,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":106,"certified":true,"suppressed":false,"headroom":0.822735415089175,"rig_ceiling":5384.126984126984,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":77.828125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":142.8125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":139.71484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":143.05859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":143.05078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":129,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.17067641454619376,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.1148098714721732,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":42163,"first_disqualified_conc":2,"lower_bound":false,"samples":182},{"bound_ms":100,"rps":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":42163,"first_disqualified_conc":2,"lower_bound":false,"samples":182},{"bound_ms":null,"rps":{"value":38,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2936546,"first_disqualified_conc":32,"lower_bound":false,"samples":229}],"sample_counts":{"gateway_c1_samples":97,"direct_c1_samples":264624,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":115,"401":5,"500":4,"503":4},"stream_errors_no_response":0},{"conc":96,"http_status_counts":{"200":96},"stream_errors_no_response":0},{"conc":112,"http_status_counts":{"200":108,"401":3,"503":1},"stream_errors_no_response":0},{"conc":104,"http_status_counts":{"200":104},"stream_errors_no_response":0},{"conc":108,"http_status_counts":{"200":100,"401":5,"500":1,"503":2},"stream_errors_no_response":0},{"conc":106,"http_status_counts":{"200":106},"stream_errors_no_response":0},{"conc":107,"http_status_counts":{"200":101,"401":5,"500":1},"stream_errors_no_response":0},{"conc":106,"http_status_counts":{"200":106},"stream_errors_no_response":0},{"conc":106,"http_status_counts":{"200":105,"401":1},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:openai-responses:anthropic","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:cohere:anthropic","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:gemini:anthropic","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:bedrock:anthropic","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:anthropic:anthropic","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:openai-responses:openai","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:cohere:openai","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:openai:openai","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":300,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":34017,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":589609,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":589637,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"the c=8 cost window had 1 failure(s) alongside 55 success(es); CPU divided by only the successes would describe the failures, not the work","state":"not_measured","section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"the c=8 cost window had 1 failure(s) alongside 55 success(es); CPU divided by only the successes would describe the failures, not the work","state":"not_measured","section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"the c=8 cost window had 1 failure(s) alongside 55 success(es); CPU divided by only the successes would describe the failures, not the work","state":"not_measured","section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"the c=8 cost window had 1 failure(s) alongside 55 success(es); CPU divided by only the successes would describe the failures, not the work","state":"not_measured","section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"the c=8 cost window had 1 failure(s) alongside 55 success(es); CPU divided by only the successes would describe the failures, not the work","state":"not_measured","section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"the c=8 cost window had 1 failure(s) alongside 55 success(es); CPU divided by only the successes would describe the failures, not the work","state":"not_measured","section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"the c=8 cost window had 1 failure(s) alongside 55 success(es); CPU divided by only the successes would describe the failures, not the work","state":"not_measured","section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"the c=8 cost window had 1 failure(s) alongside 55 success(es); CPU divided by only the successes would describe the failures, not the work","state":"not_measured","section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"the c=8 cost window had 1 failure(s) alongside 55 success(es); CPU divided by only the successes would describe the failures, not the work","state":"not_measured","section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":728,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":770,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":12,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":7497.836670339175,"certified":true,"suppressed":false,"headroom":0.6930218753394484,"rig_ceiling":10819.047619047618,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":213,"certified":true,"suppressed":false,"headroom":0.6930218753394484,"rig_ceiling":10819.047619047618,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":87.15625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":141.876953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":136.578125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":142.4921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":142.47265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":134,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.1159464817427444,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":300,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-0.1148904573303369,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":32,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":34193,"first_disqualified_conc":2,"lower_bound":false,"samples":196},{"bound_ms":100,"rps":{"value":32,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":34193,"first_disqualified_conc":2,"lower_bound":false,"samples":196},{"bound_ms":null,"rps":{"value":38,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":2530662,"first_disqualified_conc":32,"lower_bound":false,"samples":229}],"sample_counts":{"gateway_c1_samples":102,"direct_c1_samples":263027,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":238,"401":8,"500":3,"503":7},"stream_errors_no_response":0},{"conc":192,"http_status_counts":{"200":192},"stream_errors_no_response":0},{"conc":224,"http_status_counts":{"200":161,"401":33,"500":17,"503":13},"stream_errors_no_response":0},{"conc":208,"http_status_counts":{"200":208},"stream_errors_no_response":0},{"conc":216,"http_status_counts":{"200":150,"401":50,"500":6,"503":10},"stream_errors_no_response":0},{"conc":212,"http_status_counts":{"200":212},"stream_errors_no_response":0},{"conc":214,"http_status_counts":{"200":131,"401":40,"500":15,"503":28},"stream_errors_no_response":0},{"conc":213,"http_status_counts":{"200":213},"stream_errors_no_response":0},{"conc":213,"http_status_counts":{"200":155,"401":37,"500":7,"503":14},"stream_errors_no_response":0},{"conc":213,"http_status_counts":{"200":213},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:anthropic:openai","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:gemini:openai","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f:bedrock:openai","gateway_id":"one-api","gateway_build":"justsong/one-api:v0.6.10@sha256:af6b361630bdf408b86fce7aa8fc131b37ea5177ae866d121ba7c002e773e477","run_id":"run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","measured_at":"2026-11-10T17:39:58Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","detail":"Pinned v0.6.10 source 3915ce9814b8261a1ab13ed93adec58b463cd75c: router/relay.go exposes OpenAI chat ingress, not native Responses, Anthropic, Gemini, Cohere or Bedrock ingress. relay/adaptor supports OpenAI, Anthropic, Gemini, Cohere (v1/chat) and AWS Bedrock (relay/adaptor/aws, InvokeModel through the AWS SDK at its default HTTPS endpoint, which the engine v1.2 catch-all DNS resolves to the mock) upstreams. This matrix describes the tested protocol routes, not all product features. Source: https://github.com/songquanpeng/one-api/blob/3915ce9814b8261a1ab13ed93adec58b463cd75c/router/relay.go","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-13.472073514277696,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":83239,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/one-api","raw_result_url":"https://onthebench.ai/api/v1/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-one-api-c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"c9427cc70e392f48beb8e68d908f8924a7330748ac291dab1cd902b3f14c542f"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:openai:bedrock","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":306,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":205998,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":212984,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":213012,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":189176.47058823527,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5.286069651741294,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.6169920805069831,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":85,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":68,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":5.141176470588236,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":598.80078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1006.28515625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1006.22265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1006.6875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1006.46875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":61,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.329296963138252,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":306,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.046660991313913644,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 50ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":100,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 100ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":null,"rps":{"value":21,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":831737,"first_disqualified_conc":256,"lower_bound":false,"samples":127}],"sample_counts":{"gateway_c1_samples":29,"direct_c1_samples":262765},"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:cohere:bedrock","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:gemini:bedrock","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:openai-responses:bedrock","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":305,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":200013,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":205956,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":205984,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":182500,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5.47945205479452,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.8586890105703885,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":116,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":19,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":68,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":4.620689655172414,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":621.60546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1055.50390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1055.875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1055.9375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1055.87109375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":66,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.07183335358262208,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":305,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.04844942521841008,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 50ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":100,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 100ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":null,"rps":{"value":20,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":1943615,"first_disqualified_conc":512,"lower_bound":false,"samples":121}],"sample_counts":{"gateway_c1_samples":30,"direct_c1_samples":265071},"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:anthropic:bedrock","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":193978,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":197949,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":197977,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":175360,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5.702554744525547,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.8601191828446305,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":20,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":68,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":5.32,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":612.45703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":967.6796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":967.421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":967.99609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":967.70703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":60,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.25204829296188547,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.05193333183537614,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 50ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":100,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 100ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":null,"rps":{"value":21,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":1585939,"first_disqualified_conc":1024,"lower_bound":false,"samples":127}],"sample_counts":{"gateway_c1_samples":31,"direct_c1_samples":259772},"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:bedrock:bedrock","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:openai:anthropic","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:openai-responses:anthropic","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:anthropic:anthropic","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":303,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":193933,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":195022,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":195049,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":181353.38345864663,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5.514096185737976,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9610214676333437,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":133,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":22,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":69,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":5.879699248120301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":153935,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":165859,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21070us) came in under the mock's (21073us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21167us) came in under the mock's (21758us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":919.9861683706454,"certified":true,"suppressed":false,"headroom":0.2786496567661089,"rig_ceiling":3301.5873015873017,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":65,"certified":true,"suppressed":false,"headroom":0.2786496567661089,"rig_ceiling":3301.5873015873017,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":593.06640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":948.5078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":948.43359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":949.0078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":948.77734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":67,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.30135980491154135,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":303,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.05169704844231903,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 50ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":100,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 100ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":null,"rps":{"value":21,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":559997,"first_disqualified_conc":512,"lower_bound":false,"samples":128}],"sample_counts":{"gateway_c1_samples":31,"direct_c1_samples":264589,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":96,"http_status_counts":{"200":96},"stream_errors_no_response":0},{"conc":80,"http_status_counts":{"200":80},"stream_errors_no_response":0},{"conc":72,"http_status_counts":{"200":72},"stream_errors_no_response":0},{"conc":68,"http_status_counts":{"200":68},"stream_errors_no_response":0},{"conc":66,"http_status_counts":{"200":66},"stream_errors_no_response":0},{"conc":65,"http_status_counts":{"200":65},"stream_errors_no_response":0},{"conc":65,"http_status_counts":{"200":65},"stream_errors_no_response":0},{"conc":65,"http_status_counts":{"200":65},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:cohere:anthropic","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:gemini:anthropic","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:bedrock:anthropic","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:anthropic:openai-responses","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:gemini:openai-responses","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:openai:openai-responses","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:openai-responses:openai-responses","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":301,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":194984,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":198964,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":198991,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":177760,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5.625562556255626,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.8678710026382068,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":20,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":68,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":4.712,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":611.9609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":1019.71484375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":1019.88671875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":1019.96875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":1019.88671875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":66,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.19858581951462687,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.04586307382763456,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 50ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":100,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 100ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":null,"rps":{"value":21,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":16,"p99_us":1695035,"first_disqualified_conc":512,"lower_bound":false,"samples":127}],"sample_counts":{"gateway_c1_samples":30,"direct_c1_samples":268348},"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:cohere:openai-responses","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:bedrock:openai-responses","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:cohere:gemini","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:openai:gemini","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:openai-responses:gemini","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:anthropic:gemini","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:gemini:gemini","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:bedrock:gemini","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:openai-responses:cohere","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:cohere:cohere","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:openai:cohere","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:gemini:cohere","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:anthropic:cohere","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:bedrock:cohere","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:openai:openai","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":305,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":198988,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":205966,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":205993,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":182230.76923076922,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5.487547488391726,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.9645138811621244,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":130,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":21,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":68,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":6.276923076923077,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":158773,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":170494,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":38,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":669.7194949393278,"certified":true,"suppressed":false,"headroom":0.4546587088488971,"rig_ceiling":1473.015873015873,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":29,"certified":true,"suppressed":false,"headroom":0.4546587088488971,"rig_ceiling":1473.015873015873,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":617.203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":971.84765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":972.3203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":972.4453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":972.31640625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":61,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.12924042130805258,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":305,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.052286481252150616,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 50ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":100,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 100ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":null,"rps":{"value":19,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":788018,"first_disqualified_conc":512,"lower_bound":false,"samples":118}],"sample_counts":{"gateway_c1_samples":30,"direct_c1_samples":266566,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":24,"http_status_counts":{"200":24},"stream_errors_no_response":0},{"conc":28,"http_status_counts":{"200":28},"stream_errors_no_response":0},{"conc":30,"http_status_counts":{"200":30},"stream_errors_no_response":0},{"conc":29,"http_status_counts":{"200":29},"stream_errors_no_response":0},{"conc":29,"http_status_counts":{"200":29},"stream_errors_no_response":0},{"conc":29,"http_status_counts":{"200":29},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:openai-responses:openai","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:bedrock:openai","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:anthropic:openai","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":304,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":195998,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":198984,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":199012,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":178977.27272727274,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":5.587301587301587,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.647250235848493,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":88,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":14,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":84,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":5.840909090909091,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":156909,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":168479,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21072us) came in under the mock's (21073us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21139us) came in under the mock's (21149us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":787.6218019669257,"certified":true,"suppressed":false,"headroom":0.329921366515401,"rig_ceiling":2387.3015873015875,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":47,"certified":true,"suppressed":false,"headroom":0.329921366515401,"rig_ceiling":2387.3015873015875,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":605.86328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":961.5234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":961.328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":961.90625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":961.73046875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":61,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.20581317972839427,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":304,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.17242880842301658,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 50ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":100,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 100ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":null,"rps":{"value":21,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":1051244,"first_disqualified_conc":256,"lower_bound":false,"samples":128}],"sample_counts":{"gateway_c1_samples":30,"direct_c1_samples":261879,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":48,"http_status_counts":{"200":48},"stream_errors_no_response":0},{"conc":40,"http_status_counts":{"200":40},"stream_errors_no_response":0},{"conc":44,"http_status_counts":{"200":44},"stream_errors_no_response":0},{"conc":46,"http_status_counts":{"200":46},"stream_errors_no_response":0},{"conc":47,"http_status_counts":{"200":47},"stream_errors_no_response":0},{"conc":47,"http_status_counts":{"200":47},"stream_errors_no_response":0},{"conc":47,"http_status_counts":{"200":47},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:cohere:openai","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e:gemini:openai","gateway_id":"plano","gateway_build":"katanemo/plano:0.4.39@sha256:35a84cb873ee5f16cded33eeda72736f90798962b7489d58650c0db7b546e9ac","run_id":"run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","measured_at":"2026-11-10T17:40:07Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","detail":"Pinned 0.4.39 source b13ddd632bde1fea7678386c345665ea5fa671af: hermesllm/src/clients/endpoints.rs accepts Chat, Responses and Messages ingress, with no Gemini, Cohere or Bedrock client API. providers/id.rs::compatible_api_for_client selects native Responses only for Responses ingress to OpenAI, Chat for Messages-to-OpenAI, native Messages for Messages-to-Anthropic, and Converse for all three ingress APIs to amazon_bedrock. Anthropic Chat/Responses ingress and Gemini provider use OpenAI-compatible upstream wire; they are not native Anthropic/Gemini translations. The normal provider mapping is retained rather than substituting a different provider for the OpenAI column. Seven supported pairs are enabled, including the documented Bedrock upstream.","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":4.480296052973523,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":100509,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/plano","raw_result_url":"https://onthebench.ai/api/v1/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-plano-0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"0a4ce97da36f11e7c407eef9280f69688b7ea1f007616709f0b914993cb8ba8e"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:bedrock:openai","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:gemini:openai","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:openai:openai","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":301,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":1196,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":3902,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":3938,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":36,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1528.3748925193465,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":654.2897327707454,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2950181957762656,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":4652,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":775,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":22,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.007308684436801376,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":28114,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":29752,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":440,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":4996.208831040471,"certified":true,"suppressed":false,"headroom":0.06475501077097384,"rig_ceiling":77155.55555555556,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1519,"certified":true,"suppressed":false,"headroom":0.06475501077097384,"rig_ceiling":77155.55555555556,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":125.0703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":245.171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":142.08203125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":250.01953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":263.984375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.20228106254272332,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-13.958014762556811,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":831,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":3723,"first_disqualified_conc":2,"lower_bound":false,"samples":4992},{"bound_ms":10,"rps":{"value":835,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":5035,"first_disqualified_conc":8,"lower_bound":false,"samples":5012},{"bound_ms":50,"rps":{"value":836,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":14740,"first_disqualified_conc":32,"lower_bound":false,"samples":5018},{"bound_ms":100,"rps":{"value":836,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":14740,"first_disqualified_conc":128,"lower_bound":false,"samples":5018},{"bound_ms":null,"rps":{"value":836,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":8,"p99_us":14740,"first_disqualified_conc":2048,"lower_bound":false,"samples":5018}],"sample_counts":{"gateway_c1_samples":4304,"direct_c1_samples":246357,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":1810},"stream_errors_no_response":238},{"conc":1536,"http_status_counts":{"200":703},"stream_errors_no_response":833},{"conc":1280,"http_status_counts":{"200":1280},"stream_errors_no_response":0},{"conc":1408,"http_status_counts":{"200":1408},"stream_errors_no_response":0},{"conc":1472,"http_status_counts":{"200":1472},"stream_errors_no_response":0},{"conc":1504,"http_status_counts":{"200":1504},"stream_errors_no_response":0},{"conc":1520,"http_status_counts":{"200":1507},"stream_errors_no_response":13},{"conc":1512,"http_status_counts":{"200":1512},"stream_errors_no_response":0},{"conc":1516,"http_status_counts":{"200":1516},"stream_errors_no_response":0},{"conc":1518,"http_status_counts":{"200":1518},"stream_errors_no_response":0},{"conc":1519,"http_status_counts":{"200":1519},"stream_errors_no_response":0},{"conc":1519,"http_status_counts":{"200":1445},"stream_errors_no_response":74},{"conc":1519,"http_status_counts":{"200":1519},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:openai-responses:openai","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:anthropic:openai","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:cohere:openai","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:gemini:cohere","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:cohere:cohere","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":301,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":1128,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":3714,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":3750,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":36,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1331.2265455263678,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":751.186943620178,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.27976268545278316,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":5063,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":843,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":22,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.004937783922575548,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":124.9921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":245.9453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":141.52734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":249.91796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":264.00390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.8062392422532753,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-14.173574778061678,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":860,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4850,"first_disqualified_conc":4,"lower_bound":false,"samples":5161},{"bound_ms":10,"rps":{"value":860,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4850,"first_disqualified_conc":8,"lower_bound":false,"samples":5161},{"bound_ms":50,"rps":{"value":860,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4850,"first_disqualified_conc":32,"lower_bound":false,"samples":5161},{"bound_ms":100,"rps":{"value":860,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4850,"first_disqualified_conc":128,"lower_bound":false,"samples":5161},{"bound_ms":null,"rps":{"value":860,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4850,"first_disqualified_conc":8192,"lower_bound":false,"samples":5161}],"sample_counts":{"gateway_c1_samples":4519,"direct_c1_samples":248527},"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:bedrock:cohere","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:openai:cohere","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"cohere","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":301,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":1204,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":3792,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":3819,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1383.828045035824,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":722.6331360946746,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2804933340659576,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":4885,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":813,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":22,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.006345957011258956,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer cohere with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":124.9375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":244.7734375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":141.5234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":250.15625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":264.8359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.6758821056788261,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-14.251585141872061,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":821,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4986,"first_disqualified_conc":4,"lower_bound":false,"samples":4929},{"bound_ms":10,"rps":{"value":828,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":7637,"first_disqualified_conc":8,"lower_bound":false,"samples":4975},{"bound_ms":50,"rps":{"value":828,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":7637,"first_disqualified_conc":32,"lower_bound":false,"samples":4975},{"bound_ms":100,"rps":{"value":828,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":7637,"first_disqualified_conc":64,"lower_bound":false,"samples":4975},{"bound_ms":null,"rps":{"value":828,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":7637,"first_disqualified_conc":2048,"lower_bound":false,"samples":4975}],"sample_counts":{"gateway_c1_samples":4255,"direct_c1_samples":267416},"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:openai-responses:cohere","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:anthropic:cohere","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:gemini:anthropic","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:cohere:anthropic","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:bedrock:anthropic","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:anthropic:anthropic","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":301,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":1196,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":3745,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":3781,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":36,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1549.1750589243625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":645.5048409405256,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.3000290843961638,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":4667,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":777,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":22,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.006642382686950932,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":28106,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":29379,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":7,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":400,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":4968.797812465507,"certified":true,"suppressed":false,"headroom":0.08999375062825637,"rig_ceiling":55212.69841269841,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1087,"certified":true,"suppressed":false,"headroom":0.08999375062825637,"rig_ceiling":55212.69841269841,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":125.01171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":245.30859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":141.72265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":253.70703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":268.375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-1.4567790319639966,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-14.091683426869364,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":823,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4974,"first_disqualified_conc":4,"lower_bound":false,"samples":4941},{"bound_ms":10,"rps":{"value":823,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4974,"first_disqualified_conc":8,"lower_bound":false,"samples":4941},{"bound_ms":50,"rps":{"value":823,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4974,"first_disqualified_conc":32,"lower_bound":false,"samples":4941},{"bound_ms":100,"rps":{"value":823,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4974,"first_disqualified_conc":64,"lower_bound":false,"samples":4941},{"bound_ms":null,"rps":{"value":823,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4974,"first_disqualified_conc":4096,"lower_bound":false,"samples":4941}],"sample_counts":{"gateway_c1_samples":4332,"direct_c1_samples":243420,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":1798},"stream_errors_no_response":250},{"conc":1536,"http_status_counts":{"200":1151},"stream_errors_no_response":385},{"conc":1280,"http_status_counts":{"200":918},"stream_errors_no_response":362},{"conc":1152,"http_status_counts":{"200":919},"stream_errors_no_response":233},{"conc":1088,"http_status_counts":{"200":1088},"stream_errors_no_response":0},{"conc":1056,"http_status_counts":{"200":1056},"stream_errors_no_response":0},{"conc":1072,"http_status_counts":{"200":1072},"stream_errors_no_response":0},{"conc":1080,"http_status_counts":{"200":1080},"stream_errors_no_response":0},{"conc":1084,"http_status_counts":{"200":1084},"stream_errors_no_response":0},{"conc":1086,"http_status_counts":{"200":1086},"stream_errors_no_response":0},{"conc":1087,"http_status_counts":{"200":1087},"stream_errors_no_response":0},{"conc":1087,"http_status_counts":{"200":1087},"stream_errors_no_response":0},{"conc":1087,"http_status_counts":{"200":1087},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:openai:anthropic","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":301,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":1238,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":3814,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":3850,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":36,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1583.7204158372042,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":631.4245810055866,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.29703957089791033,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":4521,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":753,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":22,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.007078080070780801,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":28321,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":29941,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":2,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":363,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":4586.904652284963,"certified":true,"suppressed":false,"headroom":0.08559685814394333,"rig_ceiling":53587.30158730159,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1055,"certified":true,"suppressed":false,"headroom":0.08559685814394333,"rig_ceiling":53587.30158730159,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":125.16796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":246.1328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":141.94921875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":250.5703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":265.05078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":1.690937807428462,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-13.914391835390527,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":824,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4965,"first_disqualified_conc":4,"lower_bound":false,"samples":4948},{"bound_ms":10,"rps":{"value":824,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4965,"first_disqualified_conc":8,"lower_bound":false,"samples":4948},{"bound_ms":50,"rps":{"value":824,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4965,"first_disqualified_conc":32,"lower_bound":false,"samples":4948},{"bound_ms":100,"rps":{"value":824,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4965,"first_disqualified_conc":64,"lower_bound":false,"samples":4948},{"bound_ms":null,"rps":{"value":824,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4965,"first_disqualified_conc":1024,"lower_bound":false,"samples":4948}],"sample_counts":{"gateway_c1_samples":4169,"direct_c1_samples":245970,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":1626},"stream_errors_no_response":422},{"conc":1536,"http_status_counts":{"200":1211},"stream_errors_no_response":325},{"conc":1280,"http_status_counts":{"200":854},"stream_errors_no_response":426},{"conc":1152,"http_status_counts":{"200":950},"stream_errors_no_response":202},{"conc":1088,"http_status_counts":{"200":724},"stream_errors_no_response":364},{"conc":1056,"http_status_counts":{"200":1050},"stream_errors_no_response":6},{"conc":1040,"http_status_counts":{"200":1040},"stream_errors_no_response":0},{"conc":1048,"http_status_counts":{"200":1048},"stream_errors_no_response":0},{"conc":1052,"http_status_counts":{"200":1052},"stream_errors_no_response":0},{"conc":1054,"http_status_counts":{"200":1054},"stream_errors_no_response":0},{"conc":1055,"http_status_counts":{"200":1055},"stream_errors_no_response":0},{"conc":1055,"http_status_counts":{"200":1055},"stream_errors_no_response":0},{"conc":1055,"http_status_counts":{"200":1055},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:openai-responses:anthropic","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:gemini:bedrock","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:bedrock:bedrock","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":301,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":1316,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":4150,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":4186,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":36,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1534.8314606741574,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":651.5373352855052,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.28341991320088467,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":4450,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":741,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":22,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.00651685393258427,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":125.0234375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":248.47265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":142.32421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":253.05859375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":268.09765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0.32678350093744585,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-14.193759845506435,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":757,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4901,"first_disqualified_conc":4,"lower_bound":false,"samples":4547},{"bound_ms":10,"rps":{"value":757,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4901,"first_disqualified_conc":8,"lower_bound":false,"samples":4547},{"bound_ms":50,"rps":{"value":757,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4901,"first_disqualified_conc":32,"lower_bound":false,"samples":4547},{"bound_ms":100,"rps":{"value":757,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4901,"first_disqualified_conc":64,"lower_bound":false,"samples":4547},{"bound_ms":null,"rps":{"value":757,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4901,"first_disqualified_conc":2048,"lower_bound":false,"samples":4547}],"sample_counts":{"gateway_c1_samples":3896,"direct_c1_samples":249657},"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:openai-responses:bedrock","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:anthropic:bedrock","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":301,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":1438,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":4467,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":4494,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1632.6039913440732,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":612.5184094256259,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2817735890792074,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":4159,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":693,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":22,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.006732387593171436,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":125.078125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":250.26953125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":142.53125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":254.4296875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":268.28125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-2.6607292570829726,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-13.92566222223078,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":703,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":3631,"first_disqualified_conc":2,"lower_bound":false,"samples":4223},{"bound_ms":10,"rps":{"value":706,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":5153,"first_disqualified_conc":8,"lower_bound":false,"samples":4237},{"bound_ms":50,"rps":{"value":706,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":5153,"first_disqualified_conc":32,"lower_bound":false,"samples":4237},{"bound_ms":100,"rps":{"value":706,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":5153,"first_disqualified_conc":64,"lower_bound":false,"samples":4237},{"bound_ms":null,"rps":{"value":706,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":5153,"first_disqualified_conc":2048,"lower_bound":false,"samples":4237}],"sample_counts":{"gateway_c1_samples":3579,"direct_c1_samples":262193},"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:cohere:bedrock","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:openai:bedrock","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":301,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":1430,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":4577,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":4604,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1664.2264519277696,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":600.8797653958944,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.28285540553056554,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":4098,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":682,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":22,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.008296730112249878,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":125.18359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":248.43359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":142.8125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":253.90625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":268.37890625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-1.4534577210595634,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-14.021685221981265,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":685,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":3665,"first_disqualified_conc":2,"lower_bound":false,"samples":4114},{"bound_ms":10,"rps":{"value":696,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":5158,"first_disqualified_conc":8,"lower_bound":false,"samples":4177},{"bound_ms":50,"rps":{"value":696,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":5158,"first_disqualified_conc":32,"lower_bound":false,"samples":4177},{"bound_ms":100,"rps":{"value":696,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":5158,"first_disqualified_conc":64,"lower_bound":false,"samples":4177},{"bound_ms":null,"rps":{"value":696,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":5158,"first_disqualified_conc":2048,"lower_bound":false,"samples":4177}],"sample_counts":{"gateway_c1_samples":3563,"direct_c1_samples":269629},"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:openai-responses:gemini","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:anthropic:gemini","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:gemini:gemini","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":301,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":1145,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":4018,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":4054,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":36,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1361.8368962787015,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":734.3023255813954,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2855655650654452,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":5052,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":841,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":22,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.007719714964370546,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":124.1015625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":245.73828125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":139.9609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":249.89453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":263.9375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-0.8131857410506222,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-14.014769680173412,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":865,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4925,"first_disqualified_conc":4,"lower_bound":false,"samples":5193},{"bound_ms":10,"rps":{"value":868,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":7392,"first_disqualified_conc":8,"lower_bound":false,"samples":5212},{"bound_ms":50,"rps":{"value":868,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":7392,"first_disqualified_conc":32,"lower_bound":false,"samples":5212},{"bound_ms":100,"rps":{"value":868,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":7392,"first_disqualified_conc":128,"lower_bound":false,"samples":5212},{"bound_ms":null,"rps":{"value":868,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":4,"p99_us":7392,"first_disqualified_conc":8192,"lower_bound":false,"samples":5212}],"sample_counts":{"gateway_c1_samples":4389,"direct_c1_samples":249981},"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:openai:gemini","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"gemini","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":301,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":1199,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":3969,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":4005,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":36,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1420.5373880441575,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":703.9589442815249,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2829118072434173,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":4801,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":800,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":22,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.006040408248281608,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":125.1796875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":247.013671875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":142.2265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":250.59765625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":265.39453125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-1.3601061120886508,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-14.117312549343774,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":800,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1,"p99_us":3598,"first_disqualified_conc":2,"lower_bound":false,"samples":4805},{"bound_ms":10,"rps":{"value":809,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":5011,"first_disqualified_conc":8,"lower_bound":false,"samples":4857},{"bound_ms":50,"rps":{"value":809,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":5011,"first_disqualified_conc":32,"lower_bound":false,"samples":4857},{"bound_ms":100,"rps":{"value":809,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":5011,"first_disqualified_conc":64,"lower_bound":false,"samples":4857},{"bound_ms":null,"rps":{"value":809,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":5011,"first_disqualified_conc":2048,"lower_bound":false,"samples":4857}],"sample_counts":{"gateway_c1_samples":4241,"direct_c1_samples":249376},"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:bedrock:gemini","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:cohere:gemini","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:anthropic:openai-responses","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:openai-responses:openai-responses","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":301,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":1238,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":3854,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":3881,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":27,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":1427.6795005202914,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":700.4373177842565,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.2847080853636584,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":4805,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":800,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":22,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0.004162330905306972,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":125.03125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":243.68359375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":141.9609375,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":249.5,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":264.3125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"memory was still oscillating with no net trend - it returns to where it was, which is not a leak when the 300s load window closed, so it has no time-to-plateau; its rate is published beside this","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":-1.9357944151524555,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":301,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":-1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":-14.082313638424981,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":809,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4982,"first_disqualified_conc":4,"lower_bound":false,"samples":4855},{"bound_ms":10,"rps":{"value":809,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4982,"first_disqualified_conc":8,"lower_bound":false,"samples":4855},{"bound_ms":50,"rps":{"value":809,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4982,"first_disqualified_conc":32,"lower_bound":false,"samples":4855},{"bound_ms":100,"rps":{"value":809,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4982,"first_disqualified_conc":64,"lower_bound":false,"samples":4855},{"bound_ms":null,"rps":{"value":809,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2,"p99_us":4982,"first_disqualified_conc":4096,"lower_bound":false,"samples":4855}],"sample_counts":{"gateway_c1_samples":4169,"direct_c1_samples":265251},"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:openai:openai-responses","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:gemini:openai-responses","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:bedrock:openai-responses","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc:cohere:openai-responses","gateway_id":"portkey","gateway_build":"portkeyai/gateway:1.15.2@sha256:9065e15701d5f90ae1728c57369f8cec137d8c7dffcd027a402a67e64f27b9d3","run_id":"run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","measured_at":"2026-11-10T17:39:55Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","detail":"Pinned v1.15.2 source ca7712947ec8c0887803990b746740d36011c697: src/index.ts registers chat-completions, messages, responses and raw proxy routes; provider operation tables determine translations. Recorder preflight proves 11 exact protocol pairs. Raw proxy passes native requests through without cross-protocol conversion, so only its matching diagonal is enabled. OpenAI Chat Completions and Responses share a provider but remain distinct upstream operations. Evidence: results/diagnostics/refresh-configuration/portkey-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-5.966797991663115,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":90459,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/portkey","raw_result_url":"https://onthebench.ai/api/v1/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-portkey-670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"670dce412ba19689b30f84f64cb8b9c31fc07125970bd210617247c56c23bfcc"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:bedrock:openai","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:gemini:openai","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:anthropic:openai","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:openai:openai","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":40976,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":41966,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":41996,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":30,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":459.0984974958264,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":2178.181818181818,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.022719677197451632,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":1198,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":199,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":6,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":707,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":746,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21071us) came in under the mock's (21072us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21132us) came in under the mock's (21137us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":8048.563390252819,"certified":true,"suppressed":false,"headroom":0.07958618369944871,"rig_ceiling":101130.15873015873,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":1991,"certified":true,"suppressed":false,"headroom":0.07958618369944871,"rig_ceiling":101130.15873015873,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":48.75390625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":67.6171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":67.6171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":67.6171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":67.6171875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":60,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.024793797064739817,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":6275,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":44012,"first_disqualified_conc":512,"lower_bound":false,"samples":37658},{"bound_ms":100,"rps":{"value":11879,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":512,"p99_us":53050,"first_disqualified_conc":1024,"lower_bound":false,"samples":71288},{"bound_ms":null,"rps":{"value":12324,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1024,"p99_us":109973,"first_disqualified_conc":null,"lower_bound":false,"samples":73956}],"sample_counts":{"gateway_c1_samples":150,"direct_c1_samples":253662,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1792,"http_status_counts":{"200":1792},"stream_errors_no_response":0},{"conc":1920,"http_status_counts":{"200":1920},"stream_errors_no_response":0},{"conc":1984,"http_status_counts":{"200":1984},"stream_errors_no_response":0},{"conc":2016,"http_status_counts":{"200":1921},"stream_errors_no_response":95},{"conc":2000,"http_status_counts":{"200":2000},"stream_errors_no_response":0},{"conc":1992,"http_status_counts":{"200":1992},"stream_errors_no_response":0},{"conc":1988,"http_status_counts":{"200":1988},"stream_errors_no_response":0},{"conc":1990,"http_status_counts":{"200":1990},"stream_errors_no_response":0},{"conc":1991,"http_status_counts":{"200":1991},"stream_errors_no_response":0},{"conc":1991,"http_status_counts":{"200":1991},"stream_errors_no_response":0},{"conc":1991,"http_status_counts":{"200":1991},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:openai-responses:openai","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:cohere:openai","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:openai-responses:gemini","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:bedrock:gemini","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:openai:gemini","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"gemini","upstream_transport":"tls","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":40971,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":40996,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":41035,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":39,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":578.1758957654723,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":1729.5774647887324,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.02929945152374234,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":1228,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":204,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":6,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer gemini with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":50.7421875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":67.95703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":67.95703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":67.95703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":67.95703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":66,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.02471548973980937,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":6222,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":46028,"first_disqualified_conc":512,"lower_bound":false,"samples":37338},{"bound_ms":100,"rps":{"value":13987,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2048,"p99_us":89992,"first_disqualified_conc":4096,"lower_bound":false,"samples":83936},{"bound_ms":null,"rps":{"value":13987,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":2048,"p99_us":89992,"first_disqualified_conc":null,"lower_bound":false,"samples":83936}],"sample_counts":{"gateway_c1_samples":148,"direct_c1_samples":210647},"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:gemini:gemini","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:anthropic:gemini","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:cohere:gemini","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"gemini","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:gemini:bedrock","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:cohere:bedrock","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:anthropic:bedrock","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:bedrock:bedrock","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:openai-responses:bedrock","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"bedrock","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:openai:bedrock","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"bedrock","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":40977,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":41012,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":41040,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":468.6192468619247,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":2133.928571428571,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.02312914819538198,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":1195,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":199,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":6,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer bedrock with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":50.73828125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":67.5546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":67.5546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":67.5546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":67.5546875,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":72,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.02471213797252101,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":6243,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":44045,"first_disqualified_conc":512,"lower_bound":false,"samples":37466},{"bound_ms":100,"rps":{"value":11572,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":512,"p99_us":56041,"first_disqualified_conc":1024,"lower_bound":false,"samples":69447},{"bound_ms":null,"rps":{"value":11572,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":512,"p99_us":56041,"first_disqualified_conc":null,"lower_bound":false,"samples":69447}],"sample_counts":{"gateway_c1_samples":148,"direct_c1_samples":261020},"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:openai:openai-responses","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"openai-responses","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":40976,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":40988,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":41017,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":29,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":452.2613065326633,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":2211.111111111111,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.022306389008679293,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":1194,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":198,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":6,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":null,"certified":false,"suppressed":false,"reason":"untestable","detail":"the mock does not answer openai-responses with a native event stream, so the rig cannot pose the streaming question here","state":"untestable","section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":50.70703125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":67.7265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":67.7265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":67.7265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":67.7265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":78,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.024782967493901836,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":6238,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":44025,"first_disqualified_conc":512,"lower_bound":false,"samples":37439},{"bound_ms":100,"rps":{"value":11643,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":512,"p99_us":54915,"first_disqualified_conc":1024,"lower_bound":false,"samples":69870},{"bound_ms":null,"rps":{"value":11643,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":512,"p99_us":54915,"first_disqualified_conc":null,"lower_bound":false,"samples":69870}],"sample_counts":{"gateway_c1_samples":148,"direct_c1_samples":258742},"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:gemini:openai-responses","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:cohere:openai-responses","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:openai-responses:openai-responses","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:bedrock:openai-responses","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:anthropic:openai-responses","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"openai-responses","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:anthropic:anthropic","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:cohere:anthropic","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:openai-responses:anthropic","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:gemini:anthropic","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:bedrock:anthropic","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"anthropic","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:openai:anthropic","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"served","raw_state":true,"reason":null,"detail":null,"evidence_scope":"Recorded run evidence","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"anthropic","upstream_transport":"http","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":"cold restart, 60s idle read at rest, then load at c=32 for 300s, then 60s with the load removed. `plateaued` is a verdict over the trailing 60s, taken at the END of the load: steady means drift under 1% and range under 2% of the mean. The verdict does not end the load - every cell is loaded for the full duration.","memory_concurrency":null,"memory_load_s":302,"cost_concurrency":8,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{"added_latency_p50_us":{"value":40977,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"added_latency_p99_us":{"value":41418,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"gateway_c1_p99_us":{"value":41446,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"direct_c1_p99_us":{"value":28,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole."},"cpu_us_per_request":{"value":460.63651591289783,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"rps_per_cpu_second":{"value":2170.9090909090905,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_conc":{"value":8,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_core_utilisation":{"value":0.02271457592885274,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_ok":{"value":1194,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_window_rps":{"value":198,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_threads":{"value":6,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_nonvol_ctxt_per_request":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"cost_majflt":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"perf","definition":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else."},"added_ttft_p50_us":{"value":780,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_ttft_p99_us":{"value":856,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p50_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21071us) came in under the mock's (21076us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"added_gap_p99_us":{"value":null,"certified":false,"suppressed":false,"reason":"below_resolution","detail":"the gateway's own inter-frame gap at this percentile (21138us) came in under the mock's (21216us), which a proxy cannot really do - the added gap here is below what this rig can resolve against the mock's 20ms pacing","state":"below_resolution","section":"stream","definition":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added."},"streams_sustained_fps":{"value":8248.116178163695,"certified":true,"suppressed":false,"headroom":0.08086891795697099,"rig_ceiling":101993.6507936508,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"streams_sustained":{"value":2008,"certified":true,"suppressed":false,"headroom":0.08086891795697099,"rig_ceiling":101993.6507936508,"state":"valid","reason":null,"section":"stream","definition":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"idle_rss_mib":{"value":50.7265625,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"steady_state_rss_mib":{"value":67.6328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"recovered_rss_mib":{"value":67.6328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_mib":{"value":67.6328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"peak_rss_hwm_mib":{"value":67.6328125,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"time_to_plateau_s":{"value":90,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"growth_rate_mib_per_min":{"value":0,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"load_s":{"value":302,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_shape":{"value":null,"certified":false,"suppressed":false,"reason":"not_measured","detail":"a settled idle window has no unsettled shape to describe","state":"not_measured","section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_static":{"value":1,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."},"idle_growth_rate_mib_per_min":{"value":0.024722742409502847,"certified":true,"suppressed":false,"state":"valid","reason":null,"section":"memory","definition":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran."}},"frontier":[{"bound_ms":1,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 1ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":5,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 5ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":10,"rps":{"value":null,"certified":false,"suppressed":false,"reason":"bound_not_met","detail":"every cleanly-served rung that reported a tail had it at or above 10ms, so this gateway carried no measurable throughput under that bound","state":"bound_not_met"},"concurrency":null,"p99_us":null,"first_disqualified_conc":null,"lower_bound":false},{"bound_ms":50,"rps":{"value":6258,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":256,"p99_us":44016,"first_disqualified_conc":512,"lower_bound":false,"samples":37557},{"bound_ms":100,"rps":{"value":11681,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":512,"p99_us":53969,"first_disqualified_conc":1024,"lower_bound":false,"samples":70099},{"bound_ms":null,"rps":{"value":12514,"certified":true,"suppressed":false,"state":"valid","reason":null},"concurrency":1024,"p99_us":108004,"first_disqualified_conc":null,"lower_bound":false,"samples":75094}],"sample_counts":{"gateway_c1_samples":151,"direct_c1_samples":259563,"ttft_gw_samples":100,"ttft_direct_samples":100,"gap_gw_samples":63,"gap_direct_samples":63},"stream_rung_diagnostics":[{"conc":1,"http_status_counts":{"200":1},"stream_errors_no_response":0},{"conc":2,"http_status_counts":{"200":2},"stream_errors_no_response":0},{"conc":4,"http_status_counts":{"200":4},"stream_errors_no_response":0},{"conc":8,"http_status_counts":{"200":8},"stream_errors_no_response":0},{"conc":16,"http_status_counts":{"200":16},"stream_errors_no_response":0},{"conc":32,"http_status_counts":{"200":32},"stream_errors_no_response":0},{"conc":64,"http_status_counts":{"200":64},"stream_errors_no_response":0},{"conc":128,"http_status_counts":{"200":128},"stream_errors_no_response":0},{"conc":256,"http_status_counts":{"200":256},"stream_errors_no_response":0},{"conc":512,"http_status_counts":{"200":512},"stream_errors_no_response":0},{"conc":1024,"http_status_counts":{"200":1024},"stream_errors_no_response":0},{"conc":2048,"http_status_counts":{"200":2048},"stream_errors_no_response":0},{"conc":1536,"http_status_counts":{"200":1536},"stream_errors_no_response":0},{"conc":1792,"http_status_counts":{"200":1792},"stream_errors_no_response":0},{"conc":1920,"http_status_counts":{"200":1920},"stream_errors_no_response":0},{"conc":1984,"http_status_counts":{"200":1984},"stream_errors_no_response":0},{"conc":2016,"http_status_counts":{"200":2016},"stream_errors_no_response":0},{"conc":2000,"http_status_counts":{"200":2000},"stream_errors_no_response":0},{"conc":2008,"http_status_counts":{"200":2008},"stream_errors_no_response":0},{"conc":2012,"http_status_counts":{"200":2012},"stream_errors_no_response":0},{"conc":2010,"http_status_counts":{"200":2010},"stream_errors_no_response":0},{"conc":2009,"http_status_counts":{"200":2009},"stream_errors_no_response":0},{"conc":2008,"http_status_counts":{"200":2008},"stream_errors_no_response":0},{"conc":2008,"http_status_counts":{"200":2008},"stream_errors_no_response":0}],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:openai-responses:cohere","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai-responses","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:openai:cohere","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"openai","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:anthropic:cohere","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"anthropic","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:cohere:cohere","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"cohere","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:bedrock:cohere","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"bedrock","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"},{"schema_version":"1.0","measurement_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1:gemini:cohere","gateway_id":"tensorzero","gateway_build":"tensorzero/gateway:2026.6.0@sha256:9133135ccdaa23c0fffc2965f20b2a89b077ea3ad9a86a6f15b2a641bff67c89","run_id":"run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","measured_at":"2026-11-10T17:40:04Z","status":{"state":"not_declared","raw_state":"not_configurable","reason":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","detail":"Pinned 2026.6.0 source 62eb8f63e8ec62018d70420dbf1a8c5d1c026315: crates/tensorzero-core/src/endpoints/openai_compatible/mod.rs exposes Chat Completions ingress; native TensorZero inference is outside this six-wire matrix. Four configured upstream adapters passed response-content and exact upstream-recorder checks: OpenAI Chat Completions, OpenAI Responses, Anthropic and Bedrock Converse. Gemini (google_ai_studio_gemini) hardcodes generativelanguage.googleapis.com over HTTPS; the engine v1.2 catch-all DNS resolves it to the mock, so that cell is probed and carries upstream_transport \"tls\". Evidence: results/diagnostics/refresh-configuration/tensorzero-protocol-preflight.json","evidence_scope":"Historical capability research, not a fresh probe of the tested build. Referenced versions may differ from gateway_build; do not infer current capability.","probe_http_status":null,"response_excerpt":null},"test":{"ingress_protocol":"gemini","upstream_protocol":"cohere","hardware":"AWS m7g.4xlarge (Graviton3, 16 cores / 64 GB). Gateway-under-test pinned to 4 cores (the comparable basis); mock and load generator on 6 cores each so the mock never bottlenecks the streaming sweep. Ubuntu 24.04. One dedicated box per gateway.","architecture":"arm64","harness_commit":"d240badd966933fa19eec0606b2d1f3945f18ad2","harness_dirty":false,"mock_sha256":"99d01f709d8d9c1f49055aa265d5eb7475264d6b513911fe317aee74a71190f8","load_generator_sha256":null,"qualification":{"band_pct":20,"baseline_rps":96199,"baseline_samples":1,"concurrency":32,"drift_pct":-7.312965831245647,"observed_absent_detail":null,"observed_absent_reason":null,"observed_rps":89164,"outcome":"pass"},"definitions":{"memory":"See `matrix.memory.protocol` and each cell's `memory.protocol`, which state the durations and both plateau thresholds formatted from the constants that ran.","perf.added_latency":"WHAT THE GATEWAY ADDS, at concurrency 1. The gateway leg's percentile minus the same request taken straight to the mock, one at a time, both legs in the same window. No search and no threshold: there is nothing to stop early and nothing to decide. A difference at or below what the rig can resolve publishes `below_resolution` - the best result the comparison can express - and every surface renders that apart from a never-measured hole.","perf.cost":"WHAT A REQUEST COST, in microseconds of gateway CPU. User plus system time, summed across the gateway's whole PROCESS TREE (a gateway that forks workers keeps its CPU in the children, and a parent-only reading would report a busy gateway as idle), sampled as the DIFFERENCE across one load window and divided by the requests that window actually completed. WHY IT EXISTS BESIDE THROUGHPUT. Peak rps is a SATURATION number: once a gateway fills the cores it is pinned to, the ladder stops describing the gateway and starts describing the box, and two gateways at that wall read the same however different they are. Cost per request has no such ceiling - at saturation both serve the same rate by definition, and the one doing less work per request still reads lower. It is also the figure that maps to money, since half the CPU serves the same traffic on half the instance. MEASURED AT ONE CONCURRENCY, c=8, HELD IDENTICAL FOR EVERY GATEWAY and published as `cost_window_conc`. No single concurrency is below saturation for a field spanning double-digit to five-figure rps, so matched LOAD is impossible and matched CONCURRENCY is the honest substitute - and it is stated rather than assumed. REFUSED, NOT ESTIMATED, IN TWO CASES. A window with ANY failure publishes no cost at all: CPU spent refusing requests is real, but dividing it by only the successes would describe the failures rather than the work. And a window with major page faults means the box was SWAPPING, so what was timed is the disk - those numbers publish with `cost_majflt` non-zero and the cost itself marked a harness fault, so a reader sees why the row looks wrong instead of finding a hole. `cost_core_utilisation` is the fraction of the PINNED cores busy across that same window. It is what makes a peak interpretable: near 1.0 the gateway had filled the cores it was given and its throughput number is a real ceiling; well below it, the limit is somewhere else and the peak means something else.","perf.frontier":"THROUGHPUT AT A TAIL LATENCY YOU ACCEPT. For each declared bound, the most requests/sec this cell carried while 99% of requests finished under that bound AND it failed none it accepted. Bounds: 1ms, 5ms, 10ms, 50ms, 100ms plus one reading with no latency bound at all (failures only). ONE MEASUREMENT, READ SIX WAYS: a single concurrency sweep, published as `sweep_max_proxy`, so every reading is re-derivable from the rungs rather than taken on trust. Monotone non-decreasing across the bounds BY CONSTRUCTION - relaxing a bound only adds rungs to the set a maximum is taken over - so a reading cannot exceed a looser one. ZERO FAILURES, not a tolerance: the load generator counts connections the RIG could not open separately (`rig_refused`, and those windows are discarded), so a failure reaching a reading is the gateway losing a request it had accepted. TERMINATION: the sweep climbs a doubling ladder from the floor and stops at the first concurrency where no window served cleanly, because more concurrency cannot un-fail those requests. Each reading names `first_disqualified_conc`, the lowest concurrency above it that stopped qualifying - that pair is the proof it is a boundary. When nothing above it disqualified, the sweep ran out of range instead of finding a limit, and the reading sets `lower_bound: true`: the rate is real, it is simply a floor rather than a ceiling. A window in which the gateway completed no request at all is a failed rung like any other. Only a RIG stop (a thread or connection this host could not make, or a load generator with no readable result) interrupts the sweep; its frontier is then read from the rungs below the stop, every reading sets `lower_bound: true`, and the absences name the rung and the cause. A bound no rung met publishes `bound_not_met`. The `p99_us` on a reading is the tail the winning rung ACTUALLY produced, never the bound - 4ms under a 100ms bound is not the same finding as 99ms - and `samples` is the number of completed requests it was taken over.","stream.added_ttft_and_gap":"WHAT THE GATEWAY ADDS TO A STREAM, at concurrency 1: time to the first event, and the gaps between frames inside one stream, each as the gateway leg minus the same stream taken straight to the mock. Time-to-first-EVENT rather than first token, deliberately and by the same definition on both legs: a frame budget of one is satisfied by a dialect's opening scaffolding (openai sends a role delta, anthropic a `message_start`), so the quantity is the same on both sides and the difference still isolates what the gateway added.","stream.streams_sustained":"THE MOST CONCURRENT SSE STREAMS THIS CELL CARRIES CLEANLY, and the frames/sec at that point. A concurrency holds the gate when every expected content frame arrives (ratio 1.0, i.e. no tolerance - a proxy that drops a frame has dropped a user's token), no inter-frame gap exceeds 10x the mock's own pacing interval, and under 0.1% of streams error. TERMINATION: the gate is monotone in concurrency, so the ceiling is a boundary found by bisection - a pass proven at n and a failure measured above it. `sweep_streams` carries every rung probed. A range whose top still passed publishes an absence rather than a ceiling, because the top of the range is our choice and not the gateway's answer."},"sweep_duration_s":null,"sweep_ttft_ms":null,"memory_protocol":null,"memory_concurrency":null,"memory_load_s":null,"cost_concurrency":null,"somaxconn":65535,"tcp_max_syn_backlog":65535},"metrics":{},"frontier":[],"provenance":{"source_url":"https://onthebench.ai/gateways/tensorzero","raw_result_url":"https://onthebench.ai/api/v1/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1/raw.json","run_url":"https://onthebench.ai/gateways/runs/run-tensorzero-e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1","methodology_url":"https://onthebench.ai/gateways/method","raw_sha256":"e1970318de8934076256f5f37dbf5d7fb6019f1dd5bb339bd959ac71a0a1a8e1"},"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"published_at":"2026-11-14T11:09:23.855Z","generated_at":"2026-11-14T11:09:24.858Z","age_days":3,"age_as_of":"2026-11-14T11:09:24.858Z","age":"3 days old at collection generation"}],"disclosure":{"operator":"Busbar team","operator_is_entrant":true},"comparison_policy":{"id":"matched-cell-v1","description":"Ratios require an identical protocol pair, hardware, architecture, metric definition, clean harness commit, recorded mock digest, and metric-specific workload. Missing evidence never matches missing evidence. These are observed gateway measurements against a local mock, not a claim about model latency or statistical significance."},"dataset_id":"dataset-c7c493902ee4473d65d1886e06f885af4adac2f4f5a4ea5fc70f2efa5f230c2f","generated_at":"2026-11-14T11:09:24.858Z","published_at":"2026-11-14T11:09:23.855Z","measured_range":{"earliest":"2026-11-10T17:39:19Z","latest":"2026-11-11T03:17:59Z"},"metadata_as_of":"2026-11-14","license":"https://creativecommons.org/licenses/by/4.0/","publication_time_scope":"Release timestamp recorded when publishing begins; not the first historical publication of these measurements.","archive_history_included":true,"source_url":"https://onthebench.ai/gateways","immutable_url":"https://onthebench.ai/api/v1/benchmarks/dataset-c7c493902ee4473d65d1886e06f885af4adac2f4f5a4ea5fc70f2efa5f230c2f","gateways":[{"id":"agentgateway","name":"agentgateway","url":"https://onthebench.ai/gateways/agentgateway","json_url":"https://onthebench.ai/api/v1/gateways/agentgateway","current_metadata":{"stars":6183,"stars_as_of":"2026-11-14","first_commit":"2025-03-18"}},{"id":"aisix","name":"AISIX (api7)","url":"https://onthebench.ai/gateways/aisix","json_url":"https://onthebench.ai/api/v1/gateways/aisix","current_metadata":{"stars":219,"stars_as_of":"2026-11-14","first_commit":"2026-04-17"}},{"id":"apisix","name":"APISIX","url":"https://onthebench.ai/gateways/apisix","json_url":"https://onthebench.ai/api/v1/gateways/apisix","current_metadata":{"stars":17321,"stars_as_of":"2026-11-14","first_commit":"2019-04-10"}},{"id":"bifrost","name":"Bifrost","url":"https://onthebench.ai/gateways/bifrost","json_url":"https://onthebench.ai/api/v1/gateways/bifrost","current_metadata":{"stars":9550,"stars_as_of":"2026-11-14","first_commit":"2025-03-19"}},{"id":"busbar","name":"Busbar","url":"https://onthebench.ai/gateways/busbar","json_url":"https://onthebench.ai/api/v1/gateways/busbar","current_metadata":{"stars":221,"stars_as_of":"2026-11-14","first_commit":"2026-05-29"}},{"id":"gomodel","name":"GoModel","url":"https://onthebench.ai/gateways/gomodel","json_url":"https://onthebench.ai/api/v1/gateways/gomodel","current_metadata":{"stars":1321,"stars_as_of":"2026-11-14","first_commit":"2025-12-13"}},{"id":"helicone","name":"Helicone","url":"https://onthebench.ai/gateways/helicone","json_url":"https://onthebench.ai/api/v1/gateways/helicone","current_metadata":{"stars":643,"stars_as_of":"2026-11-14","first_commit":"2025-04-01"}},{"id":"higress","name":"Higress","url":"https://onthebench.ai/gateways/higress","json_url":"https://onthebench.ai/api/v1/gateways/higress","current_metadata":{"stars":9764,"stars_as_of":"2026-11-14","first_commit":"2022-10-27"}},{"id":"kong","name":"Kong","url":"https://onthebench.ai/gateways/kong","json_url":"https://onthebench.ai/api/v1/gateways/kong","current_metadata":{"stars":44378,"stars_as_of":"2026-11-14","first_commit":"2014-11-17"}},{"id":"litellm-python","name":"LiteLLM · Python","url":"https://onthebench.ai/gateways/litellm-python","json_url":"https://onthebench.ai/api/v1/gateways/litellm-python","current_metadata":{"stars":63241,"stars_as_of":"2026-11-14","first_commit":"2023-07-27"}},{"id":"litellm-rust","name":"LiteLLM · Rust","url":"https://onthebench.ai/gateways/litellm-rust","json_url":"https://onthebench.ai/api/v1/gateways/litellm-rust","current_metadata":{"stars":63241,"stars_as_of":"2026-11-14","first_commit":"2023-07-27"}},{"id":"one-api","name":"One-API","url":"https://onthebench.ai/gateways/one-api","json_url":"https://onthebench.ai/api/v1/gateways/one-api","current_metadata":{"stars":37431,"stars_as_of":"2026-11-14","first_commit":"2023-04-22"}},{"id":"plano","name":"Plano","url":"https://onthebench.ai/gateways/plano","json_url":"https://onthebench.ai/api/v1/gateways/plano","current_metadata":{"stars":7117,"stars_as_of":"2026-11-14","first_commit":"2024-07-09"}},{"id":"portkey","name":"Portkey","url":"https://onthebench.ai/gateways/portkey","json_url":"https://onthebench.ai/api/v1/gateways/portkey","current_metadata":{"stars":13377,"stars_as_of":"2026-11-14","first_commit":"2023-08-23"}},{"id":"tensorzero","name":"TensorZero","url":"https://onthebench.ai/gateways/tensorzero","json_url":"https://onthebench.ai/api/v1/gateways/tensorzero","current_metadata":{"stars":11701,"stars_as_of":"2026-11-14","first_commit":"2024-07-16"}}],"metrics":["added_gap_p50_us","added_gap_p99_us","added_latency_p50_us","added_latency_p99_us","added_ttft_p50_us","added_ttft_p99_us","cost_core_utilisation","cost_majflt","cost_nonvol_ctxt_per_request","cost_threads","cost_window_conc","cost_window_ok","cost_window_rps","cpu_us_per_request","direct_c1_p99_us","gateway_c1_p99_us","growth_rate_mib_per_min","idle_growth_rate_mib_per_min","idle_rss_mib","idle_shape","idle_static","load_s","peak_rss_hwm_mib","peak_rss_mib","recovered_rss_mib","rps_per_cpu_second","shape","steady_state_rss_mib","streams_sustained","streams_sustained_fps","throughput_rps","time_to_plateau_s"]}
